jonashaag
commented
1 year ago Lots of packages from Anaconda channels are being replaced with conda-forge ones in your command. That could cause problems.
Open mfansler opened 1 year ago
jonashaag
commented
1 year ago Lots of packages from Anaconda channels are being replaced with conda-forge ones in your command. That could cause problems.
mfansler
commented
1 year ago @jonashaag I've thought about channel switching, but I can't get that to stick. Maybe you can elaborate if I missed something.
I'm aware that mixing channels can lead to runtime issues, such as missing symbol references, but that's not the problem here, rather the transactions are warning/failing during file copying/link creation. Perhaps if the failure was in some pre-link or post-link script.
Another possibility is clobbering, say if one channel's build includes files deployed by a differently-named package in another build. But that's not what fails in the openssl case, rather the UNLINK transaction cryptically fails to remove files, then the LINK dies when it tries to create an already existing softlink.
Moreover, in the follow-up test, the exact same transaction pair that emits a libmamba critical error, namely,
defaults::openssl-1.1.1q-h7f8727e_0conda-forge::openssl-1.1.1s-h0b41bf4_1works without any error. So, I don't see anything inherent to the channel switching that can explain the behavior.
I'm more thinking there is something about how the Anaconda base is installed, e.g., the Docker image runs these commands after installation:
find /opt/conda/ -follow -type f -name '*.a' -delete
find /opt/conda/ -follow -type f -name '*.js.map' -delete
/opt/conda/bin/conda clean -afyDoes Micromamba somehow depend on something being deleted there? Does it short-circuit if a single file listed in the conda-meta/*.json manifests isn't there?
jonashaag
commented
1 year ago Maybe you can try to do the package removals only and have a look at the filesystem state after that? Are some files left over? Maybe the json files are incomplete?
jonashaag
commented
1 year ago Another wild guess, maybe Mamba doesn’t correctly handle packages being switched from one channel to another (my idea would be that it doesn’t correctly remove the old package). I guess this could also be checked by doing the uninstalls explicitly and inspecting the filesystem.
mfansler
commented
1 year ago Maybe you can try to do the package removals only and have a look at the filesystem state after that? Are some files left over? Maybe the json files are incomplete?
Let's try force-removing openssl.
(base) root@f653a4a3b93e:/# micromamba remove --dry-run -fn base --json openssl
{
"actions": {
"PREFIX": "/opt/conda",
"UNLINK": [
{
"build": "h7f8727e_0",
"build_number": 0,
"build_string": "h7f8727e_0",
"channel": "pkgs/main",
"constrains": null,
"depends": [
"ca-certificates",
"libgcc-ng >=7.5.0"
],
"fn": "openssl-1.1.1q-h7f8727e_0.conda",
"license": "",
"md5": "2ac47797afee2ece8d339c18b095b8d8",
"name": "openssl",
"sha256": "49804293b87141523b2606836ece8e2aaa5202983698fd91e7c36bdb8c8a8de5",
"size": 2649280,
"subdir": "linux-64",
"timestamp": 0,
"track_features": "",
"url": "https://repo.anaconda.com/pkgs/main/linux-64/openssl-1.1.1q-h7f8727e_0.conda",
"version": "1.1.1q"
}
]
},
"dry_run": true,
"prefix": "/opt/conda",
"success": true
}Here it reports the error in missing the *.a file, but says it will be ignored.
(base) root@f653a4a3b93e:/# micromamba remove -fn base -vv openssl
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
debug libmamba Loading configuration
debug libmamba 107 configurables computed
info libmamba Loading single package record: "/opt/conda/conda-meta/mkl_random-1.2.2-py39h51133e4_0.json"
info libmamba Loading single package record: "/opt/conda/conda-meta/textdistance-4.2.1-pyhd3eb1b0_0.json"
info libmamba Loading single package record: "/opt/conda/conda-meta/python-fastjsonschema-2.16.2-py39h06a4308_0.json"
[ omitting additional packages ]
info libmamba Loading single package record: "/opt/conda/conda-meta/openssl-1.1.1q-h7f8727e_0.json"
[ omitting additional packages ]
info libmamba Loading single package record: "/opt/conda/conda-meta/libev-4.33-h7f8727e_1.json"
info libmamba Loading single package record: "/opt/conda/conda-meta/zope-1.0-py39h06a4308_1.json"
info libmamba Loading single package record: "/opt/conda/conda-meta/conda-package-handling-1.9.0-py39h5eee18b_0.json"
info libmamba Adding package record to repo _ipyw_jlab_nb_ext_conf
info libmamba Adding package record to repo _libgcc_mutex
info libmamba Adding package record to repo _openmp_mutex
[ omitting additional packages ]
info libmamba Adding package record to repo openssl
[ omitting additional packages ]
info libmamba Adding package record to repo zope.interface
info libmamba Adding package record to repo zstd
info libmamba Parsing MatchSpec openssl
info libmamba Found python in installed packages 3.9.13
Transaction
Prefix: /opt/conda
Removing specs:
- openssl
debug libmamba Verify cache '/opt/conda/pkgs' for package extracted directory 'openssl-1.1.1q-h7f8727e_0'
debug libmamba Extracted package cache '/opt/conda/pkgs/openssl-1.1.1q-h7f8727e_0' not found
debug libmamba 'openssl-1.1.1q-h7f8727e_0' extracted directory cache is invalid
debug libmamba Verify cache '/opt/conda/pkgs' for package tarball 'openssl-1.1.1q-h7f8727e_0'
debug libmamba 'openssl-1.1.1q-h7f8727e_0' tarball cache is invalid
Package Version Build Channel Size
──────────────────────────────────────────────────────
Remove:
──────────────────────────────────────────────────────
- openssl 1.1.1q h7f8727e_0 pkgs/main 3MB
Summary:
Remove: 1 packages
Total download: 0 B
──────────────────────────────────────────────────────
Confirm changes: [Y/n] y
debug libmamba Locking directory '/opt/conda/conda-meta'
debug libmamba Successfully locked
info libmamba Cleaned 0 .mamba_trash files. 0 remaining.
Transaction starting
info libmamba All targets to download are cached
Unlinking openssl-1.1.1q-h7f8727e_0
info libmamba Unlinking package 'openssl-1.1.1q-h7f8727e_0'
debug libmamba Use metadata found at '/opt/conda/conda-meta/openssl-1.1.1q-h7f8727e_0.json'
debug libmamba Error when removing file '/opt/conda/lib/libcrypto.a' will be ignored
info libmamba Waiting for pyc compilation to finish
Transaction finished
info libmamba Opening history file: "/opt/conda/conda-meta/history"
debug libmamba Unlocking '/opt/conda/conda-meta'
info libmamba Freeing transaction.
info libmamba Freeing pool.Inspecting the filesystem with ls after the above transaction indicates that most files listed in the JSON were not deleted, but some were. Also, the conda-meta/openssl-1.1.1q-h7f8727e_0.json was deleted.
(base) root@f653a4a3b93e:/# cd /opt/conda
(base) root@f653a4a3b93e:/opt/conda# ls bin/{c_rehash,openssl}
ls: cannot access 'bin/c_rehash': No such file or directory
bin/openssl
(base) root@f653a4a3b93e:/opt/conda# ls include/openssl
aes.h bnerr.h conf_api.h dtls1.h evperr.h obj_mac.h pkcs12.h rsa.h stack.h x509.h
asn1.h buffer.h conferr.h e_os2.h hmac.h objects.h pkcs12err.h rsaerr.h store.h x509_vfy.h
asn1_mac.h buffererr.h crypto.h ebcdic.h idea.h objectserr.h pkcs7.h safestack.h storeerr.h x509err.h
asn1err.h camellia.h cryptoerr.h ec.h kdf.h ocsp.h pkcs7err.h seed.h symhacks.h x509v3.h
asn1t.h cast.h ct.h ecdh.h kdferr.h ocsperr.h rand.h sha.h tls1.h x509v3err.h
async.h cmac.h cterr.h ecdsa.h lhash.h opensslconf.h rand_drbg.h srp.h ts.h
asyncerr.h cms.h des.h ecerr.h md2.h opensslv.h randerr.h srtp.h tserr.h
bio.h cmserr.h dh.h engine.h md4.h ossl_typ.h rc2.h ssl.h txt_db.h
bioerr.h comp.h dherr.h engineerr.h md5.h pem.h rc4.h ssl2.h ui.h
blowfish.h comperr.h dsa.h err.h mdc2.h pem2.h rc5.h ssl3.h uierr.h
bn.h conf.h dsaerr.h evp.h modes.h pemerr.h ripemd.h sslerr.h whrlpool.h
(base) root@f653a4a3b93e:/opt/conda# ls lib/engines-1.1/
afalg.so capi.so padlock.so
(base) root@f653a4a3b93e:/opt/conda# ls lib/libcrypto*
lib/libcrypto.so
(base) root@f653a4a3b93e:/opt/conda# ls lib/libssl*
lib/libssl.so lib/libssl.so.1.1 lib/libssl3.so
(base) root@f653a4a3b93e:/opt/conda# ls lib/pkgconfig/{libcrypto,libssl,openssl}.pc
ls: cannot access 'lib/pkgconfig/libcrypto.pc': No such file or directory
ls: cannot access 'lib/pkgconfig/libssl.pc': No such file or directory
ls: cannot access 'lib/pkgconfig/openssl.pc': No such file or directory
(base) root@f653a4a3b93e:/opt/conda# ls ssl
cacert.pem cert.pem ct_log_list.cnf ct_log_list.cnf.dist misc openssl.cnf openssl.cnf.dist
(base) root@f653a4a3b93e:/opt/conda# ls ssl/misc/
CA.pl tsget tsget.plThe fact that some files are deleted seems to imply that libmamba is getting interrupted (possibly by the missing file error), and then fails to resume the deletion task. I.e., this is a bug.
jonashaag
commented
1 year ago Can you please run with another -v (TRACE)? From the code it doesn’t look like it’s possible that inability to remove a file will stop the rest of the unlinking
mfansler
commented
1 year ago Can you please run with another
-v(TRACE)? From the code it doesn’t look like it’s possible that inability to remove a file will stop the rest of the unlinking
Yes, seems like it continues to unlink some files. Here's the last part with TRACE-level logging:
debug libmamba Locking directory '/opt/conda/conda-meta'
trace libmamba Lockfile created at '/opt/conda/conda-meta/conda-meta.lock'
debug libmamba Successfully locked
info libmamba Cleaned 0 .mamba_trash files. 0 remaining.
Transaction starting
info libmamba All targets to download are cached
Unlinking openssl-1.1.1q-h7f8727e_0
info libmamba Unlinking package 'openssl-1.1.1q-h7f8727e_0'
debug libmamba Use metadata found at '/opt/conda/conda-meta/openssl-1.1.1q-h7f8727e_0.json'
trace libmamba Unlinking '/opt/conda/bin/c_rehash'
trace libmamba Unlinking '/opt/conda/lib/libcrypto.a'
debug libmamba Error when removing file '/opt/conda/lib/libcrypto.a' will be ignored
trace libmamba Unlinking '/opt/conda/lib/libcrypto.so.1.1'
trace libmamba Unlinking '/opt/conda/lib/pkgconfig/libcrypto.pc'
trace libmamba Unlinking '/opt/conda/lib/pkgconfig/libssl.pc'
trace libmamba Unlinking '/opt/conda/lib/pkgconfig/openssl.pc'
info libmamba Waiting for pyc compilation to finish
Transaction finished
info libmamba Opening history file: "/opt/conda/conda-meta/history"
debug libmamba Unlocking '/opt/conda/conda-meta'
trace libmamba Removing file '/opt/conda/conda-meta/conda-meta.lock'
info libmamba Freeing transaction.
info libmamba Freeing pool.Looking at the full openssl-1.1.1q-h7f8727e_0.json file, the pattern we observe is that only files listed under paths_data.paths are deleted. The remaining files are those that are only listed under the files key.
I'll have to check on the specification, but is this something malformed with this JSON shipping from Anaconda? Checking the one installed in my own system's base (conda-forge/osx-64::openssl-3.0.7-hfd90126_1.json), it appears to have a paths_data.paths entry for every object.
jonashaag
commented
1 year ago Maybe this is a recent extension to the file format that we don’t implement yet
mfansler
commented
1 year ago Just another note that I've also tested the Miniconda3 image (continuumio/miniconda3:22.11.1) and it has similar issues.
In summary, the finding is that Anaconda and Miniconda ship with JSON files in the conda-meta that do not include paths_data.paths entries for every file that is installed. Micromamba apparently depends on this information, but it would be more robust to also check the files list object in the JSON.
The reason why my original "follow-up" test passed was that installing the same package from Anaconda Cloud does result in a full version of the JSON, i.e., includes paths_data.paths entries for all files. In fact, I am unable to find an independent source for a version of the package that has this abridged version of the JSON. Downloading directly from the URL listed in the JSON results in a .conda package that includes all the paths information. Perhaps it is something particular to conda-constructor (assuming that's how they package Anaconda/Miniconda)?
Not sure where to report this, but here's some curious behavior: Attempting to use Micromamba to install
mambainto the Anaconda installation in the official Docker image (continuumio/anaconda3) results in warnings and critical errors fromlibmamba. These seem to indicate cryptic failures in unlinking existing packages.I was hoping to give users a recommendation for how to quickly install Mamba into Anaconda base by using Micromamba, but this is worrisome behavior. Please let me know if something more needs to be configured besides the
MAMBA_ROOT_PREFIX.Reproducible Results
Additional Notes
This only appears to impact Micromamba when operating on the base.
Non-base works fine
Mimicking the same transaction in a new environment does not lead to an issue: