Using gzip and protocol=4 in ORviewer

[taldcroft]

@jskrist - here is what you should do in ORviewer for pickling the catalogs:

>>> import pickle
>>> import gzip
>>> from proseco import get_aca_catalog

>>> aca = get_aca_catalog(20603)

>>> pickle.dump(aca, open('aca1.pkl', 'wb'))

>>> with gzip.open('aca1.pkl.gz', 'wb') as fh:
...     pickle.dump(aca, fh, protocol=4)
...     

>>> ls -l aca1.pkl*
-rw-r--r--  1 aldcroft  staff  171939 Feb 20 17:37 aca1.pkl
-rw-r--r--  1 aldcroft  staff   22239 Feb 20 17:37 aca1.pkl.gz

>>> with gzip.open('aca1.pkl.gz', 'rb') as fh:
...     aca1 = pickle.load(fh)
...     

[jeanconn]

Great. I'll include this in my interim suggestions for the pickle-write function (which may or may not be replaced with a call from the new proseco/sparkles calling code in the next release).

On Wed, Feb 20, 2019, 5:43 PM Tom Aldcroft notifications@github.com wrote:

@jskrist https://github.com/jskrist - here is what you should do in ORviewer for pickling the catalogs:

import pickle import gzip from proseco import get_aca_catalog

aca = get_aca_catalog(20603)

pickle.dump(aca, open('aca1.pkl', 'wb'))

with gzip.open('aca1.pkl.gz', 'wb') as fh: ... pickle.dump(aca, fh, protocol=4) ...

ls -l aca1.pkl* -rw-r--r-- 1 aldcroft staff 171939 Feb 20 17:37 aca1.pkl -rw-r--r-- 1 aldcroft staff 22239 Feb 20 17:37 aca1.pkl.gz

with gzip.open('aca1.pkl.gz', 'rb') as fh: ... aca1 = pickle.load(fh) ...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sot/proseco/issues/284, or mute the thread https://github.com/notifications/unsubscribe-auth/AAwT1ElajBCZCvLPLWqSBhiYKCeJGGiwks5vPc97gaJpZM4bGRP4 .

[jeanconn]

Oh, as getting the multiline statement will take me more pyexec wrangling, what are the issues with

pickle.dump(aca, gzip.open('aca1.pkl.gz', 'wb'), protocol=4)

?

[taldcroft]

That version basically relies on garbage collection running to finally close the file handle. I think that in practice it is going to be fine, but not really ideal for production code.

[taldcroft]

For a one liner I would prefer:

with gzip.open('aca1.pkl.gz', 'wb') as fh: pickle.dump(aca, fh, protocol=4)

I think this will work. This is just using deprecated syntax, but solid file handling.

[jeanconn]

OK. I'll either close explicitly, use "with" in multiline, or use your "with" in one-liner.

[jskrist]

@jeanconn -

Oh, as getting the multiline statement will take me more pyexec wrangling, [...]

I have a MATLAB example script showing various approaches to performing multiline statements with pyexec, I'll send it your way.

[jeanconn]

I think this was done.