I ran out of resources (urllib.error.HTTPError: HTTP Error 403: rate limit exceeded)

[marekjez86]

I ran about 350 merges ... tried to merge everything non-LGC with both Sans and Serif... after a while I started to get "urllib.error.HTTPError: HTTP Error 403: rate limit exceeded" .... What do we need to do to avoid it?

====== SANS ZanabazarSquare

['NotoSans', 'NotoSansZanabazarSquare']

INFO: Download begin

Traceback (most recent call last):

File "./notobuilderCLI.py", line 883, in

sys.exit(main())

File "./notobuilderCLI.py", line 876, in main

subs

File "./notobuilderCLI.py", line 259, in init

self.buildRepoName()

File "./notobuilderCLI.py", line 358, in buildRepoName

Download(self.repoNames, self.scriptsFolder, self.hinted)

File "./notobuilderCLI.py", line 47, in init

self.dwnldSources()

File "./notobuilderCLI.py", line 70, in dwnldSources

response = urllib.request.urlretrieve(api_url)

File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 247, in urlretrieve

with contextlib.closing(urlopen(url, data)) as fp:

File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen

return opener.open(url, data, timeout)

File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open

response = meth(req, response)

File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response

'http', request, response, code, msg, hdrs)

File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 569, in error

return self._call_chain(*args)

File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain

result = func(*args)

File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 649, in http_error_default

raise HTTPError(req.full_url, code, msg, hdrs, fp)

urllib.error.HTTPError: HTTP Error 403: rate limit exceeded

[davelab6]

I'm surprised this isn't operating 100% locally

[jbmorizot]

The command line build the repo name then download the fonts from github. It allows people to always work with the last build of the fonts, and it's quite fast since they are not ufo or glyphs sources. I'm surprised github puts limits on compiled files download.

So I could add an option to pass a folder path to the command line. The class will check first it it founds the fonts folder in it, but it will require the folders are named exactly like the github repositories.

[jbmorizot]

Maybe a better idea would be to add an option to not delete the fonts folder after download. The scripts will check if they are still here before starting the download. I add it. It works with the --keep argument.

[davelab6]

If you use a local git repo you can fetch the head commit id and then tell the user to pull it it doesn't match? :)

On Tue, Aug 18, 2020, 12:54 PM jbmorizot notifications@github.com wrote:

Maybe a better idea would be to add an option to not delete the fonts folder after download. The scripts will check if they are still here before starting the download. I add it. It works with the --keep argument.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/notofonts/noto-build/issues/15#issuecomment-675595029, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB73S6F2H2IBFHCE2QX57TSBKW37ANCNFSM4QCRXXDQ .

[marekjez86]

test to see if we can handle resources

for i in Adlam AdlamUnjoined Ahom AnatolianHieroglyphs Arabic Armenian Avestan Balinese Bamum BassaVah Batak Bengali Bhaiksuki Brahmi Buginese Buhid CanadianAboriginal Carian CaucasianAlbanian Chakma Cham Cherokee Coptic Cuneiform Cypriot Deseret Devanagari Display Dogra Duployan EgyptianHieroglyphs Elbasan Elymaic Ethiopic Georgian Glagolitic Gothic Grantha Gujarati GunjalaGondi Gurmukhi HanifiRohingya Hanunoo Hatran Hebrew ImperialAramaic IndicSiyaqNumbers InscriptionalPahlavi InscriptionalParthian Javanese Kaithi Kannada KayahLi Kharoshthi Khmer Khojki Khudawadi KufiArabic Lao Lepcha Limbu LinearA LinearB Lisu Lycian Lydian Mahajani Malayalam Mandaic Manichaean Marchen MasaramGondi Math MayanNumerals MeeteiMayek MendeKikakui Meroitic Miao Modi Mongolian Mono Mro Multani Music Myanmar NKo Nabataean NaskhArabic NastaliqUrdu NewTaiLue Newa Nushu NyiakengPuachueHmong Ogham OlChiki OldHungarian OldItalic OldNorthArabian OldPermic OldPersian OldSogdian OldSouthArabian OldTurkic Oriya Osage Osmanya PahawhHmong Palmyrene PauCinHau PhagsPa Phoenician PsalterPahlavi RashiHebrew Rejang Runic Samaritan Saurashtra Sharada Shavian Siddham Sinhala Sogdian SoraSompeng Soyombo Sundanese SylotiNagri Symbols Symbols2 Syriac Tagalog Tagbanwa TaiLe TaiTham TaiViet Takri Tamil TamilSlanted TamilSupplement Tangut Telugu Thaana Thai Tibetan Tifinagh Tirhuta TraditionalNushu Ugaritic Vai Wancho WarangCiti Yi ZanabazarSquare  ; do echo "====== SANS" $i; python3 ./notobuilderCLI.py --name "NotoSans-$i" --output ttf --scripts Latin Cyrillic Greek  $i --contrast Sans --weight Thin ExtraLight Light Regular ExtraBold Bold SemiBold Medium Black --subset Full  ; echo "====== SANS" $i; python3 ./notobuilderCLI.py --name "NotoSans-$i" --output ttf --scripts Latin Cyrillic Greek  $i --contrast Serif --weight Thin ExtraLight Light Regular ExtraBold Bold SemiBold Medium Black --subset Full ; done

[jbmorizot]

I'll test it. Also I'll create a downloaded folder and add it to gitignore so one can keeps the dl fonts to avoid to dl them again and still have no modification in git. Plus I can store the SHA for each of them as Dave suggest and compare with the latest head commit SHA to see if fonts need to be pulled again. Does it sound a working plan?

Edit: it's done. Still, when you access a repo content through github API (like I did) the rate limit is 60 requests/hours. When you're using your github account, it's 5000 / hours. I'll see if it's possible to use username/password stored in git credential to have the 5000 requests limit. It makes me think a server version of the notobuilder should indeed have cloned all Noto repo, and work locally—but that is something we don't want to force on users.