search

thoth-station / package-analyzer

Analyze Python artifacts
GNU General Public License v3.0
0 stars 4 forks source link

Fetching digests for package 'selinon' in version '1.0.0' from 'https://pypi.org/simple' fails #20

Closed bissenbay closed 5 years ago

bissenbay commented 5 years ago 
---> Running application from script (app.sh) ...
2019-07-18 21:23:56,136 [1] INFO     root:119: Setting up logging to a Sentry instance 'sentry.io/1298083', environment None
2019-07-18 21:23:56,141 [1] INFO     root:150: Logging to rsyslog endpoint is turned off
2019-07-18 21:23:56,142 [1] INFO     __main__:68: Version: 0.1.0
2019-07-18 21:23:56,142 [1] INFO     thoth.digests_fetcher.python.fetch:41: Fetching digests for package 'selinon' in version '1.0.0' from 'https://pypi.org/simple' 
2019-07-18 21:23:56,856 [1] CRITICAL root:111: Traceback (most recent call last):
  File "thoth-digests-fetcher", line 133, in <module>
    cli()
  File "/opt/app-root/lib/python3.6/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/opt/app-root/lib/python3.6/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/opt/app-root/lib/python3.6/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/app-root/lib/python3.6/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/app-root/lib/python3.6/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/opt/app-root/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "thoth-digests-fetcher", line 120, in python
    result[url] = python_fetcher.fetch(package_name, package_version)
  File "/opt/app-root/src/thoth/digests_fetcher/python/fetch.py", line 44, in fetch
    files = self.source.get_package_hashes(package_name, package_version, True)
  File "/opt/app-root/lib/python3.6/site-packages/thoth/python/source.py", line 381, in get_package_hashes
    return self._warehouse_get_package_hashes(package_name, package_version, with_included_files)
  File "/opt/app-root/lib/python3.6/site-packages/thoth/python/source.py", line 191, in _warehouse_get_package_hashes
    result[-1]["contents"] = self._get_contents(item["url"], item["filename"])
  File "/opt/app-root/lib/python3.6/site-packages/thoth/python/source.py", line 172, in _get_contents
    with closing(response), ZipFile(io.BytesIO(response.content)) as zip:
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/zipfile.py", line 1108, in __init__
    self._RealGetContents()
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/zipfile.py", line 1175, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
Sentry is attempting to send 1 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
bissenbay commented 5 years ago

@fridex The reason it was failing. For these specific inputs there were downloaded 2 files.

  1. selinon-1.0.0-py3-none-any.whl
  2. selinon-1.0.0.tar.gz

To fix it:

  1. We can ignore .tar.gz files and just go with .whl files
  2. We can use gzip to extract .tar.gz files.

@fridex WDYT? Which approach is right or any suggestions?

Thanks!

bissenbay commented 5 years ago

@fridex After some research I decided to go with approach 2 because there might be cases where all packages are in .tar.gz format, so we cannot just skip them.