search

nrdg / cloudknot

A python library to run your existing code on AWS Batch
https://nrdg.github.io/cloudknot/
Other
71 stars 17 forks source link

FIX: Use default pickle protocol even for cloudpickle #273

Closed richford closed 3 years ago

richford commented 3 years ago

resolves #270 resolves #138

coveralls commented 3 years ago

Coverage Status

Coverage remained the same at 0.0% when pulling 1dd6e4e1c66c8d1fa9cc28bb1bfb081b25d53f52 on richford:fix/pickle-protocol into 8049a9f33787b4cbd9370330c13ec1066d9092d9 on nrdg:master.

arokem commented 3 years ago

Does this actually resolve the issue?

On Python 3.8: 

In [1]: import pickle

In [2]: pickle.DEFAULT_PROTOCOL
Out[2]: 4

OTOH, on Python 3.7: 

In [1]: import pickle

In [2]: pickle.DEFAULT_PROTOCOL
Out[2]: 3

Wouldn't moving from local 3.8 to execution in 3.7 still cause the same issues?

richford commented 3 years ago

You're right. Hmm. Perhaps we need to add a pickle_protocol parameter to be specified by the user.

arokem commented 3 years ago

Maybe we could set a default as a library level constant cloudkbot.DEFAULT_PICKLE_PROTOCOL to be used if user doesn’t supply an input? This could be used on both the local machine and the batch workers, regardless of discrepancies in python versions. I guess we can set this to 3 until we drop 3.7 support.

On Tue, Mar 23, 2021 at 1:58 PM Adam Richie-Halford < @.***> wrote:

You're right. Hmm. Perhaps we need to add a pickle_protocol parameter to be specified by the user.

— You are receiving this because your review was requested. Reply to this email directly, view it on GitHub https://github.com/nrdg/cloudknot/pull/273#issuecomment-805253227, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA46NWZAFZVJEMTFEPLTJTTFD6FXANCNFSM4ZV3AXRA .