libarchive.exception.ArchiveError with zip files on osx

[jschueller]

I'm updating a few recipes, some of them use zip archives and fail with the same error on osx:

Source cache directory is: /usr/local/miniconda/conda-bld/src_cache
2019-10-24T12:15:55.0700410Z INFO:conda_build.source:Source cache directory is: /usr/local/miniconda/conda-bld/src_cache
2019-10-24T12:15:55.0701750Z INFO conda_build.source:download_to_cache(43): Source cache directory is: /usr/local/miniconda/conda-bld/src_cache
2019-10-24T12:15:55.0702850Z Downloading source to cache: FMILibrary-2.0.3-src_4cc21f9e2c.zip
2019-10-24T12:15:55.0704000Z Downloading https://jmodelica.org/FMILibrary/FMILibrary-2.0.3-src.zip
2019-10-24T12:15:55.0705020Z INFO:conda_build.source:Downloading source to cache: FMILibrary-2.0.3-src_4cc21f9e2c.zip
2019-10-24T12:15:55.0706150Z INFO conda_build.source:download_to_cache(66): Downloading source to cache: FMILibrary-2.0.3-src_4cc21f9e2c.zip
2019-10-24T12:15:55.0707180Z INFO:conda_build.source:Downloading https://jmodelica.org/FMILibrary/FMILibrary-2.0.3-src.zip
2019-10-24T12:15:55.0708300Z INFO conda_build.source:download_to_cache(80): Downloading https://jmodelica.org/FMILibrary/FMILibrary-2.0.3-src.zip
2019-10-24T12:15:56.7841100Z Success
2019-10-24T12:15:56.7842020Z INFO:conda_build.source:Success
2019-10-24T12:15:56.7842580Z INFO conda_build.source:download_to_cache(91): Success
2019-10-24T12:15:56.7944540Z Extracting download
2019-10-24T12:15:56.8202280Z Traceback (most recent call last):
2019-10-24T12:15:56.8204470Z   File "/usr/local/miniconda/bin/conda-build", line 11, in <module>
2019-10-24T12:15:56.8212000Z     sys.exit(main())
2019-10-24T12:15:56.8213340Z   File "/usr/local/miniconda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 449, in main
2019-10-24T12:15:56.8236760Z     execute(sys.argv[1:])
2019-10-24T12:15:56.8238630Z   File "/usr/local/miniconda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 440, in execute
2019-10-24T12:15:56.8241790Z     verify=args.verify, variants=args.variants)
2019-10-24T12:15:56.8242990Z   File "/usr/local/miniconda/lib/python3.7/site-packages/conda_build/api.py", line 209, in build
2019-10-24T12:15:56.8245920Z     notest=notest, need_source_download=need_source_download, variants=variants)
2019-10-24T12:15:56.8247630Z   File "/usr/local/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 2339, in build_tree
2019-10-24T12:15:56.8267090Z     notest=notest,
2019-10-24T12:15:56.8268240Z   File "/usr/local/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 1413, in build
2019-10-24T12:15:56.8278460Z     try_download(m, no_download_source=False, raise_error=True)
2019-10-24T12:15:56.8279590Z   File "/usr/local/miniconda/lib/python3.7/site-packages/conda_build/render.py", line 628, in try_download
2019-10-24T12:15:56.8288110Z     source.provide(metadata)
2019-10-24T12:15:56.8290000Z   File "/usr/local/miniconda/lib/python3.7/site-packages/conda_build/source.py", line 652, in provide
2019-10-24T12:15:56.8292410Z     timeout=metadata.config.timeout, locking=metadata.config.locking)
2019-10-24T12:15:56.8293720Z   File "/usr/local/miniconda/lib/python3.7/site-packages/conda_build/source.py", line 148, in unpack
2019-10-24T12:15:56.8294290Z     tar_xf(src_path, tmpdir)
2019-10-24T12:15:56.8296070Z   File "/usr/local/miniconda/lib/python3.7/site-packages/conda_build/utils.py", line 830, in tar_xf
2019-10-24T12:15:56.8298900Z     libarchive.extract_file(tarball, flags)
2019-10-24T12:15:56.8300190Z   File "/usr/local/miniconda/lib/python3.7/site-packages/libarchive/extract.py", line 70, in extract_file
2019-10-24T12:15:56.8312250Z     extract_entries(archive, flags)
2019-10-24T12:15:56.8314150Z   File "/usr/local/miniconda/lib/python3.7/site-packages/libarchive/extract.py", line 50, in extract_entries
2019-10-24T12:15:56.8315460Z     write_header(write_p, entry._entry_p)
2019-10-24T12:15:56.8317110Z   File "/usr/local/miniconda/lib/python3.7/site-packages/libarchive/ffi.py", line 91, in check_int
2019-10-24T12:15:56.8318120Z     raise archive_error(args[0], retcode)
2019-10-24T12:15:56.8319770Z   File "/usr/local/miniconda/lib/python3.7/site-packages/libarchive/ffi.py", line 75, in archive_error
2019-10-24T12:15:56.8351190Z     raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
2019-10-24T12:15:56.8352860Z libarchive.exception.ArchiveError: Could not open FMILibrary-2.0.3 (errno=22, retcode=-30, archive_p=140490166914912)
2019-10-24T12:15:58.8114680Z ##[error]Bash exited with code '1'.
2019-10-24T12:15:58.8131850Z ##[section]Finishing: Build recipe

is it a bug of this package ?

see these prs: https://github.com/conda-forge/cminpack-feedstock/pull/8 https://github.com/conda-forge/fmilib-feedstock/pull/13

see also https://github.com/conda-forge/orekit-feedstock/pull/33

[tadeu]

I can confirm the same problem here: https://github.com/conda-forge/cgal-cpp-feedstock/pull/6

[mcs07]

Same issue for conda-forge/inchi-feedstock#2

[dpryan79]

I'm having the same issues on bioconda, reverting to version 3.3.3 fixes the issue.

[dpryan79]

I suspect the interface for the OSX library changed between 3.3.3 and 3.4.0, since this works on Linux but not OSX. That suggests that the issue is somewhere in the interface between this and python-libarchive-c.

[jjhelmus]

Should the macOS libarchive 3.4.0 packages be labeled bad until this issue can be resolved?

[jakirkham]

Labeling it broken makes sense. There will be some fallout for other packages that are linked against it. Not to say we shouldn't do it for that reason. Just trying to make people aware of this issue. If we don't do it evenly across OSes, that may also cause some surprises. Thoughts @conda-forge/core?

[CJ-Wright]

Should we pin this and downgrade the pin? (I'm thinking in the context of what levers do we have to fix this)

[jakirkham]

It doesn't appear to be pinned so that would be worth doing.

Just did a quick search and at least within conda-forge it seems like relatively few things depend on libarchive. We can probably just pull 3.4.0 across the board and rebuild anything as needed.

[CJ-Wright]

libarchive is a parent of conda-package-handling which is the parent of conda and conda-build. Is that a problem?

[jakirkham]

That appears to actually be handled by a different static build of libarchive. So I think no.

[CJ-Wright]

Ok my understanding of the path forward is to:

1. mark all 3.4.0 as broken
2. put in a pin for 3.3 into the pinning feedstock
3. run the rebuilds ourselves?

[jjhelmus]

conda no longer uses libarchive (rather that static build) but conda-build does. That said removing the 3.4.0 should not be a problem since 3.3 is still available.

[jakirkham]

Have done 1. Working on 2.

[jakirkham]

Handling 2 with PR ( https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/319 ). Please take a look.

[mdraw]

I'm having this issue with https://github.com/conda-forge/tensorboard-feedstock/pull/23, but I can't use a different archive format as a workaround (there are only .whl files). Pinning libarchive=3.3 in the host requirements for osx didn't work. Will there be a new conda-forge-pinning release soon that will fix this or is there a workaround that I can apply in the meantime?

[jakirkham]

I think more 3.4.0 builds were published at some point. We should mark them as broken too.

[jjhelmus]

I'm not seeing any 3.4.0 packages on the conda-forge channel that do not have the broken label.

[jakirkham]

I think @ocefpaf just moved them. (Thanks for doing that btw 🙂)

[jakirkham]

Closing as this was resolved a long time ago and libarchive has since upgraded past the problematic version/package