hugovk / pypi-tools

Command-line Python scripts to do things with PyPI
https://hugovk.github.io/pypi-tools
23 stars 2 forks source link

Continue if table not found #4

Closed hugovk closed 5 years ago

hugovk commented 5 years ago

There doesn't seem to be any data available for 2016-03:

$ python pypi-trends.py -p django -f 2016-01 -t 2016-04
Getting data from 1 2016 to 4 2016

2016-04-01 2016-04-30
pypinfo --start-date 2016-04-01 --end-date 2016-04-30 --percent --limit 100 --json django pyversion > data/django-2016-04.json

Traceback (most recent call last):
  File "/usr/local/bin/pypinfo", line 11, in <module>
    load_entry_point('pypinfo', 'console_scripts', 'pypinfo')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1043, in invoke
    return Command.invoke(self, ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/hugo/github/pypinfo/pypinfo/cli.py", line 161, in pypinfo
    query_rows = query_job.result(timeout=timeout // 1000)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 2501, in result
    super(QueryJob, self).result(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 672, in result
    return super(_AsyncJob, self).result(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/future/polling.py", line 120, in result
    raise self._exception
google.api_core.exceptions.BadRequest: 400 FROM clause with table wildcards matches no table
2016-03-01 2016-03-31
pypinfo --start-date 2016-03-01 --end-date 2016-03-31 --percent --limit 100 --json django pyversion > data/django-2016-03.json

Rather than exiting, print the error but continue to try the other months.

We still want to exit when quota is exceeded, as there's no point continuing. Here it shows the error for 2016-04, continues, and exits on 2016-03 due to no remaining quota:

$ python pypi-trends.py -p django -f 2016-01 -t 2016-04
Getting data from 1 2016 to 4 2016

2016-04-01 2016-04-30
pypinfo --start-date 2016-04-01 --end-date 2016-04-30 --percent --limit 100 --json django pyversion > data/django-2016-04.json

Traceback (most recent call last):
  File "/usr/local/bin/pypinfo", line 11, in <module>
    load_entry_point('pypinfo', 'console_scripts', 'pypinfo')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1043, in invoke
    return Command.invoke(self, ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/hugo/github/pypinfo/pypinfo/cli.py", line 161, in pypinfo
    query_rows = query_job.result(timeout=timeout // 1000)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 2501, in result
    super(QueryJob, self).result(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 672, in result
    return super(_AsyncJob, self).result(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/future/polling.py", line 120, in result
    raise self._exception
google.api_core.exceptions.BadRequest: 400 FROM clause with table wildcards matches no table
2016-03-01 2016-03-31
pypinfo --start-date 2016-03-01 --end-date 2016-03-31 --percent --limit 100 --json django pyversion > data/django-2016-03.json

Traceback (most recent call last):
  File "/usr/local/bin/pypinfo", line 11, in <module>
    load_entry_point('pypinfo', 'console_scripts', 'pypinfo')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1043, in invoke
    return Command.invoke(self, ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/hugo/github/pypinfo/pypinfo/cli.py", line 161, in pypinfo
    query_rows = query_job.result(timeout=timeout // 1000)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 2501, in result
    super(QueryJob, self).result(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 672, in result
    return super(_AsyncJob, self).result(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/future/polling.py", line 115, in result
    self._blocking_poll(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 2475, in _blocking_poll
    super(QueryJob, self)._blocking_poll(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/future/polling.py", line 94, in _blocking_poll
    retry_(self._done_or_raise)()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/future/polling.py", line 73, in _done_or_raise
    if not self.done():
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 2463, in done
    location=self.location)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 560, in _get_query_results
    retry, method='GET', path=path, query_params=extra_params)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 315, in _call_api
    return call()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://www.googleapis.com/bigquery/v2/projects/pypinfo-hugovk/queries/0c4b0254-ceec-4c8f-9c09-9332449fabdf?maxResults=0&timeoutMs=10000: Quota exceeded: Your project exceeded quota for free query bytes scanned. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors