Collecting package metadata (repodata.json): failed

[gaellafond]

I got the following error during the initialisation (see bellow). I'm new to python, this might be a trivial error.

I'm following the instructions described in Install locally from the Readme file:

git clone https://github.com/lsterzinger/2022-esip-kerchunk-tutorial.git

cd 2022-esip-kerchunk-tutorial

conda env create -f ./environment.yml
conda activate kerchunk

jupyter lab

It fails at the line conda env create -f ./environment.yml after several minutes.

I'm using conda 22.9.0

NOTE: I have anonymised my username with [...] in the error output.

Error output

$ conda env create -f ./environment.yml
Collecting package metadata (repodata.json): failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/urllib3/response.py", line 748, in _update_chunk_length
        self.chunk_left = int(line, 16)
    ValueError: invalid literal for int() with base 16: b''

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/urllib3/response.py", line 443, in _error_catcher
        yield
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/urllib3/response.py", line 815, in read_chunked
        self._update_chunk_length()
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/urllib3/response.py", line 752, in _update_chunk_length
        raise InvalidChunkLength(self, line)
    urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/requests/models.py", line 816, in generate
        yield from self.raw.stream(chunk_size, decode_content=True)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/urllib3/response.py", line 623, in stream
        for line in self.read_chunked(amt, decode_content=decode_content):
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/urllib3/response.py", line 844, in read_chunked
        self._original_response.close()
      File "/home/[...]/anaconda3/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/urllib3/response.py", line 460, in _error_catcher
        raise ProtocolError("Connection broken: %r" % e, e)
    urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/exceptions.py", line 1129, in __call__
        return func(*args, **kwargs)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda_env/cli/main.py", line 80, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/notices/core.py", line 72, in wrapper
        return_value = func(*args, **kwargs)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda_env/cli/main_create.py", line 156, in execute
        result[installer_type] = installer.install(prefix, pkg_specs, args, env)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda_env/installers/conda.py", line 50, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 156, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 199, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 304, in solve_final_state
        ssc = self._collect_all_metadata(ssc)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/common/io.py", line 86, in decorated
        return f(*args, **kwds)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 467, in _collect_all_metadata
        index, r = self._prepare(prepared_specs)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 1061, in _prepare
        reduced_index = get_reduced_index(self.prefix, self.channels,
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/index.py", line 301, in get_reduced_index
        new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 137, in query_all
        result = tuple(concat(executor.map(subdir_query, channel_urls)))
      File "/home/[...]/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
        yield fs.pop().result()
      File "/home/[...]/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 446, in result
        return self.__get_result()
      File "/home/[...]/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
        raise self._exception
      File "/home/[...]/anaconda3/lib/python3.9/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 129, in <lambda>
        subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 142, in query
        self.load()
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 207, in load
        _internal_state = self._load()
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 277, in _load
        raw_repodata_str = fetch_repodata_remote_request(
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 527, in fetch_repodata_remote_request
        resp = session.get(join_url(url, filename), headers=headers, proxies=session.proxies,
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
        return self.request("GET", url, **kwargs)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
        resp = self.send(prep, **send_kwargs)
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 745, in send
        r.content
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/requests/models.py", line 899, in content
        self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
      File "/home/[...]/anaconda3/lib/python3.9/site-packages/requests/models.py", line 818, in generate
        raise ChunkedEncodingError(e)
    requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

`$ /home/[...]/anaconda3/bin/conda-env create -f ./environment.yml`

  environment variables:
                 CIO_TEST=<not set>
  CONDA_AUTO_UPDATE_CONDA=false
               CONDA_ROOT=/home/[...]/anaconda3
           CURL_CA_BUNDLE=<not set>
                     PATH=/home/[...]/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/
                          usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
               WINDOWPATH=2

     active environment : None
       user config file : /home/[...]/.condarc
 populated config files : 
          conda version : 22.9.0
    conda-build version : 3.22.0
         python version : 3.9.13.final.0
       virtual packages : __linux=5.19.0=0
                          __glibc=2.35=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/[...]/anaconda3  (writable)
      conda av data dir : /home/[...]/anaconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/[...]/anaconda3/pkgs
                          /home/[...]/.conda/pkgs
       envs directories : /home/[...]/anaconda3/envs
                          /home/[...]/.conda/envs
               platform : linux-64
             user-agent : conda/22.9.0 requests/2.28.1 CPython/3.9.13 Linux/5.19.0-35-generic ubuntu/22.04.2 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

An unexpected error has occurred. Conda has prepared the above report.

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers? [y/N]: 
Timeout reached. No report sent.

[gaellafond]

I ran it again. It crashed again, but this time due to lack of RAM:

Collecting package metadata (repodata.json): done
Solving environment: \ Killed

I tried using GESIS and mybinder.org, but they both crashed with the Notebook and the Console:

Note: you may need to restart the kernel to use updated packages.

I don't know if there is a way to restart the kernel ask requested...

[lsterzinger]

Hi @gaellafond

Sorry you're having issues installing the environment. It looks like the error is likely to do with communication between your machine and Anaconda. It may be worthwile to update conda (conda update conda) or use Mamba as a drop-in replacement for conda

If you're running on GESIS/MyBinder, the memory can be an issue but if you need to restart a kernel, you can do so with Kernel (in the top menu bar) -> Restart Kernel

[gaellafond]

Thanks Lucas. I gave another try at GESIS. Once the page is loaded, I select either Notebook or Console. I type the following command:

conda env create -f ./environment.yml

It display the following logs:

Collecting package metadata (repodata.json): done
Solving environment: | ^C
failed

CondaError: KeyboardInterrupt

Note: you may need to restart the kernel to use updated packages.

NOTE: I'm NOT touching the keyboard during the whole process! That interruption CTRL+C is not coming from me.

At this point, GESIS is dead. Restarting the kernel does nothing. Closing the console and loading a new one shows a 405 Method Not Allowed error.

Either I'm doing something wrong, or GESIS can't run the code anymore.

[gaellafond]

With MyBinder, I run the same command:

conda env create -f ./environment.yml

It runs for a few minutes. The Mem status in the bottom left corner goes above 2 GB (shown in red), then jump back down to 250 MB with the following output in the console:

Collecting package metadata (repodata.json): - 
Note: you may need to restart the kernel to use updated packages.

Restarting the kernel just put me back at the beginning. The environment is reset. MyBinder clearly don't have enough RAM to run the code.

[gaellafond]

Using conda on my computer: I install conda version : 22.9.0. The update command said it updated conda to 23.1.0 When I run the command conda --version, it still shows 22.9.0

$ conda update conda
Collecting package metadata (current_repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 22.9.0
  latest version: 23.1.0

Please update conda by running

    $ conda update -n base -c defaults conda

# All requested packages already installed.

Retrieving notices: ...working... done

$ conda update -n base -c defaults conda
Collecting package metadata (current_repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 22.9.0
  latest version: 23.1.0

Please update conda by running

    $ conda update -n base -c defaults conda

# All requested packages already installed.

Retrieving notices: ...working... done

$ conda --version
conda 22.9.0

[gaellafond]

Mamba I download and installed Mamba using Mambaforge. It installed a more recent version of conda:

$ conda --version
conda 22.11.1

I ran the same command as with the others:

$ conda env create -f ./environment.yml
Collecting package metadata (repodata.json): done
Solving environment: \

It has been running for over 3 hours. It hasn't crashed yet, so I guess I'm on a good path...

[gaellafond]

I just realised that the .ipynb files are not part of the application code base. They are the pages of the Jupyter Notebook. I do not need to run the conda command lines described in the REAME when using GESIS, I can simply open those Notebook files are execute the code line by line.

I believe you assumed your users would already know that. This is my first experience with Jupyter and all I was getting was an empty Notebook or Console when i was clicking the big button in the middle of the page. I had no idea that those buttons had no use, and I had to open the .ipynb file to get somewhere.

Thanks for taking time to answer my question. Maybe you could add just a line saying to open the .ipynb files when using GESIS of MyBinder, for people like me who are coming from a different codding background.