search

dask / knit

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead
http://knit.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
53 stars 10 forks source link

More control required over construction of zip files #131

Closed jamesporritt closed 6 years ago

jamesporritt commented 6 years ago

I'd like some extra flags when creating the zipped environment to select either/both of allowZip64=True/False and compression=zipfile.ZIP_DEFLATED. I'm using the compressed zip files with spark-submit to provide the environment for my spark application. If the generated zip files are over 2Gb Spark is complaining the files have an "invalid CEN header". From what I can tell it's an issue with some parts of the Java API that don't can't handle zip64 files. This isn't knits fault - I just want a way of stepping around it.

jamesporritt commented 6 years ago

I accidentally closed this.

martindurant commented 6 years ago

Does conda-pack help for your problem? It is newer and better thought out than the code in knit.

jcrist commented 6 years ago

The code in knit is deprecated - I recommend using conda-pack instead (https://conda.github.io/conda-pack/). This works fine with spark, see https://conda.github.io/conda-pack/spark.html.