Closed NickleDave closed 2 years ago
vsoch
commented
2 years ago Row 11 of your spreadsheet is not a valid GitHub identifier:
https://github.com/sarabsethi/audioset_soundscape_feats_sethi2019/tree/master/calc_audioset_featsThat's why it says "cannot find endpoint" and shows that partial url.
NickleDave
commented
2 years ago :+1: confirmed that if I change row 11 to just https://github.com/sarabsethi/audioset_soundscape_feats_sethi2019 (without the /tree/master/calc_audioset_feats) then this row no longer throws an error
NickleDave
commented
2 years ago "valid GitHub identifier" is defined as just the landing page of a repo?
I.e. the url without anything past the project name.
just making sure I understand
NickleDave
commented
2 years ago I got further this time but got a 403, rate limit exceeded
:sob:
Is that maybe cuz I've been running it over and over again?
Or could this happen any time it runs on a sheet with greater than x rows?
INFO:rse.main.database.filesystem:github/BirdVox/birdvoxdetect was added to the the database.
INFO:rse.main.database.filesystem:github/OpenWild/caracal was added to the the database.
INFO:rse.main.database.filesystem:github/vocalpy/crowsetta was added to the the database.
INFO:rse.main.database.filesystem:github/MarineBioAcousticsRC/DetEdit was added to the the database.
INFO:rse.main.database.filesystem:github/DrCoffey/DeepSqueak was added to the the database.
INFO:rse.main.database.filesystem:github/nilomr/fieldtools was added to the the database.
INFO:rse.main.database.filesystem:github/DenaJGibbon/gibbonR-package was added to the the database.
INFO:rse.main.database.filesystem:custom/www-oldbird-org/glassofire-htm was added to the the database.
INFO:rse.main.database.filesystem:custom/www-goldwave-com was added to the the database.
ERROR:rse.utils.urls:Permission denied to query https://api.github.com/repos/Cdevenish/hardRain/topics: 403, rate limit exceeded
INFO:rse.main.database.filesystem:github/Cdevenish/hardRain was added to the the database.
INFO:rse.main.database.filesystem:custom/sites-google-com/view/alcore-suzuki/home/harkbird was added to the the database.
ERROR:rse.utils.urls:Permission denied to query https://api.github.com/repos/vocalpy/hybrid-vocal-classifier: 403, rate limit exceeded
Traceback (most recent call last):
File "/home/pimienta/Documents/repos/coding/opensci/bioacoustics/bioacoustics-software/.venv/bin/rse", line 33, in <module>
sys.exit(load_entry_point('rse', 'console_scripts', 'rse')())
File "/home/pimienta/Documents/repos/coding/opensci/bioacoustics/rse/rse/client/__init__.py", line 520, in main
main(args=args, extra=extra)
File "/home/pimienta/Documents/repos/coding/opensci/bioacoustics/rse/rse/client/imp.py", line 28, in main
importer.create(
File "/home/pimienta/Documents/repos/coding/opensci/bioacoustics/rse/rse/main/scrapers/googlesheet.py", line 99, in create
result = update_nonempty(result, data)
File "/home/pimienta/Documents/repos/coding/opensci/bioacoustics/rse/rse/utils/strings.py", line 13, in update_nonempty
for key, value in source.items():
AttributeError: 'NoneType' object has no attribute 'items'Ah yes! So this is common whenever using the GitHub api in any context, if you export GITHUB_TOKEN=xxx (as a personal access token) it should increase your limit. It only happened because you ran the same thing many times (and there are a lot of rows!)
A valid GitHub identifier matches:
https://github.com/<user-or-org>/<repo>And right now it's doing a very basic matching and using some split logic to derive the underlying name. Looking at this now, I could improve upon it - I'll open an issue for future me to work on :) The reason it's done simply is that (at least before this googlesheet addition) most GitHub urls came from places where they would always be in that format. Now that's not the case so likely my parser needs to account for that.
NickleDave
commented
2 years ago Ah yes! So this is common whenever using the GitHub api in any context, if you export GITHUB_TOKEN=xxx (as a personal access token) it should increase your limit. It only happened because you ran the same thing many times (and there are a lot of rows!)
Got it, thank you
I think I can close this then
... GitHub urls ...
So just to make sure I understand the source of the error:
this doesn't get made into a "custom" entry in the database, because some logic somewhere recognizes the "github" in the url, but then it tries to parse and fails because it's not a valid ID, and we currently require strict IDs
vsoch
commented
2 years ago Correct! And I opened an issue #77 so that you should be able to provide a poorly formatted GitHub URL and still be able to retrieve the metadata. Have your cake and eat it too :) :cake:
NickleDave
commented
2 years ago 
Excellent, will close in anticipation of cake
Description
I now have my copy of the Google sheet from @rhine3 set up so that I can start
importrunning like so:that is, I don't get any errors about missing fields, i.e. incorrect column names, now that I removed the message in the hidden row 1, and renamed the first three columns
full sheet is here: https://docs.google.com/spreadsheets/d/1Ba1MY4o5Sm1f08IekJcbxAtSjkDN71Z1RZ42kzrofJ0/edit?usp=sharing
However I do get an error now that I'm not sure how to fix, traceback below
Two notes:
What I Did
I can't see anything specifically different about the link that causes the crash
I notice that when I click on the link I get a redirect from GitHub?
But this happens for all the links in the rows above it too.