conda / conda

A system-level, binary package and environment manager running on all major operating systems and platforms.
https://docs.conda.io/projects/conda/
Other
6.33k stars 1.64k forks source link

JSONDecodeError with on-disk corrupted repodata #7190

Closed yjc765 closed 2 years ago

yjc765 commented 6 years ago

Hello, when I want to install packages there was sth wrong, as following:

Solving environment: failed

>>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

Traceback (most recent call last):
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 202, in _load
    mod_etag_headers.get('_mod'))
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 408, in fetch_repodata_remote_request
    raise Response304ContentUnchanged()
conda.core.repodata.Response304ContentUnchanged

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/exceptions.py", line 789, in __call__
    return func(*args, **kwargs)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/cli/main.py", line 78, in _main
    exit_code = do_call(args, p)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/cli/main_install.py", line 11, in execute
    install(args, parser, 'install')
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/cli/install.py", line 236, in install
    force_reinstall=context.force,
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 504, in solve_for_transaction
    force_remove, force_reinstall)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 437, in solve_for_diff
    final_precs = self.solve_final_state(deps_modifier, prune, ignore_pinned, force_remove)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 178, in solve_final_state
    index, r = self._prepare(prepared_specs)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 560, in _prepare
    self.subdirs, prepared_specs)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/index.py", line 213, in get_reduced_index
    new_records = query_all(spec)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/index.py", line 186, in query_all
    return tuple(concat(future.result() for future in as_completed(futures)))
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 90, in query
    self.load()
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 134, in load
    _internal_state = self._load()
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 208, in _load
    mod_etag_headers.get('_mod'))
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 255, in _read_local_repdata
    _internal_state = self._process_raw_repodata_str(raw_repodata_str)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 292, in _process_raw_repodata_str
    json_obj = json.loads(raw_repodata_str or '{}')
  File "/no_backup/s1258/miniconda3/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/no_backup/s1258/miniconda3/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/no_backup/s1258/miniconda3/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 86298 column 5 (char 2629609)

$ /no_backup/s1258/miniconda3/bin/conda install scikit-image

environment variables: ALSA_CONFIG_PATH=/etc/alsa-pulse.conf CIO_TEST= CONDA_ROOT=/no_backup/s1258/miniconda3 GLADE_CATALOG_PATH=:/usr/share/glade3/catalogs GLADE_MODULE_PATH=:/usr/lib64/glade3/modules GLADE_PIXMAP_PATH=:/usr/share/glade3/pixmaps LD_LIBRARY_PATH=/opt/thinlinc/lib64:/opt/thinlinc/lib MANPATH=/usr/local/man:/usr/share/man:/opt/thinlinc/share/man PATH=/no_backup/s1258/miniconda3/bin:/no_backup/s1258/miniconda3/bin:/opt/p ycharm/pycharm-community-2017.1.2/bin:/home/s1258/bin:/usr/local/bin:/ usr/bin:/bin:/usr/bin/X11:/usr/games:/opt/thinlinc/bin PULSE_RUNTIME_PATH=/var/opt/thinlinc/sessions/s1258/3/pulse PYTHONSTARTUP=/etc/pythonstart REQUESTS_CA_BUNDLE= SSL_CERT_FILE= XNLSPATH=/usr/share/X11/nls

 active environment : None
   user config file : /home/s1258/.condarc

populated config files : /home/s1258/.condarc conda version : 4.4.10 conda-build version : not installed python version : 3.6.4.final.0 base environment : /no_backup/s1258/miniconda3 (writable) channel URLs : https://repo.continuum.io/pkgs/main/linux-64 https://repo.continuum.io/pkgs/main/noarch https://repo.continuum.io/pkgs/free/linux-64 https://repo.continuum.io/pkgs/free/noarch https://repo.continuum.io/pkgs/r/linux-64 https://repo.continuum.io/pkgs/r/noarch https://repo.continuum.io/pkgs/pro/linux-64 https://repo.continuum.io/pkgs/pro/noarch package cache : /no_backup/s1258/miniconda3/pkgs /home/s1258/.conda/pkgs envs directories : /no_backup/s1258/miniconda3/envs /home/s1258/.conda/envs platform : linux-64 user-agent : conda/4.4.10 requests/2.18.4 CPython/3.6.4 Linux/4.4.62-18.6-default opensuse/42.2 glibc/2.22 UID:GID : 11258:222 netrc file : None offline mode : False

An unexpected error has occurred. Conda has prepared the above report. Upload successful.


please help me! thank you very much!

kalefranz commented 6 years ago

For your issue, use

conda clean -i

In general the code here can be refactored. The method call self._process_raw_repodata_str(raw_repodata_str) should instead take as an argument a json object that's already been loaded with json.loads(). That way if the json string is corrupted, we can do error handling that will fallback to downloading the data again.

yjc765 commented 6 years ago

dear Mr. @kalefranz

s1258@linse5:/mnt/linse8-sn/home/s1258/no_backup/s1258/miniconda3> conda clean -i s1258@linse5:/mnt/linse8-sn/home/s1258/no_backup/s1258/miniconda3>

sorry, there was no action after I typed the command above.

and when I install packages, there was still that problem, as following

s1258@linse5:/mnt/linse8-sn/home/s1258/no_backup/s1258/miniconda3> conda install scikit-image Solving environment: failed

>>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

Traceback (most recent call last):
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/exceptions.py", line 789, in __call__
    return func(*args, **kwargs)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/cli/main.py", line 78, in _main
    exit_code = do_call(args, p)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/cli/main_install.py", line 11, in execute
    install(args, parser, 'install')
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/cli/install.py", line 236, in install
    force_reinstall=context.force,
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 504, in solve_for_transaction
    force_remove, force_reinstall)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 437, in solve_for_diff
    final_precs = self.solve_final_state(deps_modifier, prune, ignore_pinned, force_remove)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 178, in solve_final_state
    index, r = self._prepare(prepared_specs)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 560, in _prepare
    self.subdirs, prepared_specs)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/index.py", line 213, in get_reduced_index
    new_records = query_all(spec)
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/index.py", line 186, in query_all
    return tuple(concat(future.result() for future in as_completed(futures)))
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 90, in query
    self.load()
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 134, in load
    _internal_state = self._load()
  File "/no_backup/s1258/miniconda3/lib/python3.6/site-packages/conda/core/repodata.py", line 215, in _load
    fh.write(raw_repodata_str or '{}')
OSError: [Errno 122] Disk quota exceeded

$ /no_backup/s1258/miniconda3/bin/conda install scikit-image

environment variables: ALSA_CONFIG_PATH=/etc/alsa-pulse.conf CIO_TEST= CONDA_ROOT=/no_backup/s1258/miniconda3 GLADE_CATALOG_PATH=:/usr/share/glade3/catalogs GLADE_MODULE_PATH=:/usr/lib64/glade3/modules GLADE_PIXMAP_PATH=:/usr/share/glade3/pixmaps LD_LIBRARY_PATH=/opt/thinlinc/lib64:/opt/thinlinc/lib MANPATH=/usr/local/man:/usr/share/man:/opt/thinlinc/share/man PATH=/no_backup/s1258/miniconda3/bin:/no_backup/s1258/miniconda3/bin:/opt/p ycharm/pycharm-community-2017.1.2/bin:/home/s1258/bin:/usr/local/bin:/ usr/bin:/bin:/usr/bin/X11:/usr/games:/opt/thinlinc/bin PULSE_RUNTIME_PATH=/var/opt/thinlinc/sessions/s1258/3/pulse PYTHONSTARTUP=/etc/pythonstart REQUESTS_CA_BUNDLE= SSL_CERT_FILE= XNLSPATH=/usr/share/X11/nls

 active environment : None
   user config file : /home/s1258/.condarc

populated config files : /home/s1258/.condarc conda version : 4.4.10 conda-build version : not installed python version : 3.6.4.final.0 base environment : /no_backup/s1258/miniconda3 (writable) channel URLs : https://repo.continuum.io/pkgs/main/linux-64 https://repo.continuum.io/pkgs/main/noarch https://repo.continuum.io/pkgs/free/linux-64 https://repo.continuum.io/pkgs/free/noarch https://repo.continuum.io/pkgs/r/linux-64 https://repo.continuum.io/pkgs/r/noarch https://repo.continuum.io/pkgs/pro/linux-64 https://repo.continuum.io/pkgs/pro/noarch package cache : /no_backup/s1258/miniconda3/pkgs /home/s1258/.conda/pkgs envs directories : /no_backup/s1258/miniconda3/envs /home/s1258/.conda/envs platform : linux-64 user-agent : conda/4.4.10 requests/2.18.4 CPython/3.6.4 Linux/4.4.62-18.6-default opensuse/42.2 glibc/2.22 UID:GID : 11258:222 netrc file : None offline mode : False

An unexpected error has occurred. Conda has prepared the above report. Upload successful.


so maybe that did not help, there is another way to solve it?

nehaljwani commented 6 years ago

OSError: [Errno 122] Disk quota exceeded

It seems that you have exceeded your disk quota. You will have to free up disk space.

kalefranz commented 6 years ago

Ahh, I missed that. Thanks @nehaljwani

kalefranz commented 6 years ago

Leaving this open still, because

In general the code here can be refactored. The method call self._process_raw_repodata_str(raw_repodata_str) should instead take as an argument a json object that's already been loaded with json.loads(). That way if the json string is corrupted, we can do error handling that will fallback to downloading the data again.

github-actions[bot] commented 2 years ago

Hi there, thank you for your contribution to Conda!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

  1. Verify that you can still reproduce the issue in the latest version of Conda

  2. Comment that the issue is still reproducible and include:

    • What version of Conda you reproduced the issue on
    • What OS and version you reproduced the issue on
    • What steps you followed to reproduce the issue
  3. It would also be helpful to have the output of the following commands available:

    • conda info
    • conda config --show-sources
    • conda list --show-channel-urls

NOTE: If this issue was closed prematurely, please leave a comment and we will gladly reopen the issue.

In case this issue was originally about a project that is covered by the Anaconda issue tracker (e.g. Anaconda, Miniconda, packages built by Anaconda, Inc. like Anaconda Navigator etc), please reopen the issue there again.

Thanks!

github-actions[bot] commented 2 years ago

Hi again!

This issue has been closed since it has not had recent activity. Please don't hesitate to leave a comment if that was done prematurely.

Thank you for your contribution.

xucaoling commented 2 years ago

When I install all packages by conda, I get this error

ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< Traceback (most recent call last): File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/subdir_data.py", line 704, in fetch_repodata_remote_request resp.raise_for_status() File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://mirrors.aliyun.com/anaconda/cloud/msys2/noarch/current_repodata.json

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1114, in call return func(*args, kwargs) File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/cli/main.py", line 86, in main_subshell exit_code = do_call(args, p) File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 90, in do_call return getattr(module, func_name)(args, parser) File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/cli/main_install.py", line 20, in execute install(args, parser, 'install') File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/cli/install.py", line 259, in install unlink_link_transaction = solver.solve_for_transaction( File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 152, in solve_for_transaction unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier, File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 195, in solve_for_diff final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned, File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 300, in solve_final_state ssc = self._collect_all_metadata(ssc) File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 86, in decorated return f(*args, *kwds) File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 463, in _collect_all_metadata index, r = self._prepare(prepared_specs) File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 1058, in _prepare reduced_index = get_reduced_index(self.prefix, self.channels, File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/index.py", line 287, in get_reduced_index new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs, File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/subdir_data.py", line 139, in query_all result = tuple(concat(executor.map(subdir_query, channel_urls))) File "/home/data/vip55/miniconda3/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator yield fs.pop().result() File "/home/data/vip55/miniconda3/lib/python3.8/concurrent/futures/_base.py", line 432, in result return self.get_result() File "/home/data/vip55/miniconda3/lib/python3.8/concurrent/futures/_base.py", line 388, in get_result raise self._exception File "/home/data/vip55/miniconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(self.args, self.kwargs) File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/subdir_data.py", line 131, in subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query( File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/subdir_data.py", line 144, in query self.load() File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/subdir_data.py", line 209, in load _internal_state = self._load() File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/subdir_data.py", line 374, in _load raw_repodata_str = fetch_repodata_remote_request( File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/core/subdir_data.py", line 735, in fetch_repodata_remote_request raise UnavailableInvalidChannel( File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/conda/exceptions.py", line 460, in init body = response.json() File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/requests/models.py", line 899, in json return complexjson.loads( File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/simplejson/init.py", line 525, in loads return _default_decoder.decode(s) File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode obj, end = self.raw_decode(s) File "/home/data/vip55/miniconda3/lib/python3.8/site-packages/simplejson/decoder.py", line 400, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) $ /home/data/vip55/miniconda3/bin/conda install -c bioconda -c conda-forge edta

environment variables: CIO_TEST= CMAKE_PREFIX_PATH=/home/data/vip55/miniconda3:/home/data/vip55/miniconda3/x86_64-conda- linux-gnu/sysroot/usr CONDA_BACKUP_JAVA_HOME= CONDA_BACKUP_JAVA_LD_LIBRARY_PATH= CONDA_BUILD_SYSROOT=/home/data/vip55/miniconda3/x86_64-conda-linux-gnu/sysroot CONDA_DEFAULT_ENV=base CONDA_EXE=/home/data/vip55/miniconda3/bin/conda CONDA_PREFIX=/home/data/vip55/miniconda3 CONDA_PROMPT_MODIFIER=(base) CONDA_PYTHON_EXE=/home/data/vip55/miniconda3/bin/python CONDA_ROOT=/home/data/vip55/miniconda3 CONDA_SHLVL=1 CURL_CA_BUNDLE= JAVA_LD_LIBRARY_PATH=/home/data/vip55/miniconda3/jre/lib/amd64/server PATH=/home/data/vip55/miniconda3/bin:/home/data/vip55/1work/dev_cell_KAS/KA S-pipe-master/scripts:/home/data/vip55/.aspera/connect/bin:/home/data/ vip55/miniconda3/bin:/home/data/vip55/miniconda3/bin:/home/data/vip55/ miniconda3/condabin:/opt/singularity/3.9.9/bin:/usr/local/sbin:/usr/lo cal/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/sna p/bin:/usr/local/go/bin:/home/data/vip55/sratoolkit.2.9.6-ubuntu64/bin :/home/data/vip55/gtfToGenePred:/home/data/vip55/.aspera/connect/bin REQUESTS_CA_BUNDLE= SSL_CERT_FILE=

active environment : base active env location : /home/data/vip55/miniconda3 shell level : 1 user config file : /home/data/vip55/.condarc populated config files : /home/data/vip55/.condarc conda version : 4.13.0 conda-build version : not installed python version : 3.8.5.final.0 virtual packages : linux=5.8.0=0 glibc=2.31=0 unix=0=0 archspec=1=x86_64 base environment : /home/data/vip55/miniconda3 (writable) conda av data dir : /home/data/vip55/miniconda3/etc/conda conda av metadata url : None channel URLs : https://conda.anaconda.org/bioconda/linux-64 https://conda.anaconda.org/bioconda/noarch https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://mirrors.aliyun.com/anaconda/pkgs/r/linux-64 https://mirrors.aliyun.com/anaconda/pkgs/r/noarch https://mirrors.aliyun.com/anaconda/pkgs/main/linux-64 https://mirrors.aliyun.com/anaconda/pkgs/main/noarch https://mirrors.aliyun.com/anaconda/cloud/bioconda/linux-64 https://mirrors.aliyun.com/anaconda/cloud/bioconda/noarch https://mirrors.aliyun.com/anaconda/cloud/msys2/linux-64 https://mirrors.aliyun.com/anaconda/cloud/msys2/noarch package cache : /home/data/vip55/miniconda3/pkgs /home/data/vip55/.conda/pkgs envs directories : /home/data/vip55/miniconda3/envs /home/data/vip55/.conda/envs platform : linux-64 user-agent : conda/4.13.0 requests/2.27.1 CPython/3.8.5 Linux/5.8.0-36-generic ubuntu/20.04.4 glibc/2.31 UID:GID : 1056:1057 netrc file : None offline mode : False

An unexpected error has occurred. Conda has prepared the above report.

Upload did not complete.

please help me! thank you very much!