medialab / minet

A webmining CLI tool & library for python.
GNU General Public License v3.0
270 stars 26 forks source link

Error on wikipedia pageviews #969

Open bmaz opened 1 month ago

bmaz commented 1 month ago
minet wikipedia pageviews pages_wikipedia -i test_wikipedia.csv --start-date 2012 --end-date 2024 > pages_test_wikipedia.csv
Collecting pageviews ━━━━━━━━━━━━ 0/2 pages ⠦ [  0%] in 510.50ms (?/s)                                                         
minet process was stopped because an error occurred!
Traceback (most recent call last):
  File "/home/bmazoyer/.pyenv/versions/minet/bin/minet", line 8, in <module>
    sys.exit(main())
  File "/home/bmazoyer/.pyenv/versions/3.8.2/envs/minet/lib/python3.8/site-packages/minet/cli/__main__.py", line 14, in main
    run("minet", __identifier__, MINET_COMMANDS)
  File "/home/bmazoyer/.pyenv/versions/3.8.2/envs/minet/lib/python3.8/site-packages/minet/cli/utils.py", line 49, in wrapper
    fn(*args, **kwargs)
  File "/home/bmazoyer/.pyenv/versions/3.8.2/envs/minet/lib/python3.8/site-packages/minet/cli/run.py", line 139, in run
    fn(cli_args)
  File "/home/bmazoyer/.pyenv/versions/3.8.2/envs/minet/lib/python3.8/site-packages/minet/cli/utils.py", line 468, in wrapper
    return action(cli_args, *args, **additional_kwargs, **kwargs)
  File "/home/bmazoyer/.pyenv/versions/3.8.2/envs/minet/lib/python3.8/site-packages/minet/cli/utils.py", line 478, in wrapper
    fn(cli_args, loading_bar=loading_bar, **kwargs)
  File "/home/bmazoyer/.pyenv/versions/3.8.2/envs/minet/lib/python3.8/site-packages/minet/cli/wikipedia/pageviews.py", line 35, in action
    for row, pageviews in client.pageviews(
  File "/home/bmazoyer/.pyenv/versions/3.8.2/envs/minet/lib/python3.8/site-packages/minet/wikipedia/wikimedia_rest_api_client.py", line 125, in pageviews
    for item in response.json()["items"]:
KeyError: 'items'
Yomguithereal commented 1 month ago

The API returns a 400 for this because the date format is incorrect. We must add some validation, or some sugar, and better document the command.