fititnt
commented
3 years ago BUG!
Ok. Descobri um problema que impede de usar o HXL Proxy (e as ferramentas por linha de comando) sem fazer download diretamente. Porém eu não estou 100% certo se isso é algo que é intencional, ao requisitar o arquivo ele é servido através da cloudflare, não tem explicitamente content-type, e tem a propriedade x-content-type-options: nosniff. Ou seja: tecnicamente o HXL Proxy (e as ferramentas de cli) estão agindo de forma correta, mas o site da https://iso639-3.sil.org/ não.
Creio que a melhor analogia seria a opção no HXL Proxy que explicitamente permite ao usuário "acessar arquivos mesmo com HTTPS inválida". De qualquer forma vou documentar aqui o problema e talvez abrir um chamado tanto no upstream como talvez contactar a iso639-3.sil.org.
Via HXL Proxy
Via cli
fititnt@bravo:~$ curl -I https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3.tab
HTTP/2 200
date: Mon, 14 Dec 2020 01:43:56 GMT
content-length: 184662
set-cookie: __cfduid=d682cecb83c4fa10d008b59ee053f7ce71607910236; expires=Wed, 13-Jan-21 01:43:56 GMT; path=/; domain=.sil.org; HttpOnly; SameSite=Lax; Secure
x-content-type-options: nosniff
last-modified: Thu, 10 Sep 2020 21:09:00 GMT
cache-control: max-age=1209600
expires: Sat, 26 Dec 2020 07:40:25 GMT
x-request-id: v-4c238bc0-3c4d-11eb-a5a6-22000a9db97e
x-ah-environment: prod
age: 151411
via: varnish
x-cache: HIT
x-cache-hits: 41
accept-ranges: bytes
cf-cache-status: DYNAMIC
cf-request-id: 07008439060000f39fba8e6000000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 60143ca1aae4f39f-GRU
fititnt@bravo:~$ hxl
hxladd hxlcount hxlexpand hxlhash hxlrename hxlsort hxlvalidate
hxlappend hxlcut hxlexplode hxlimplode hxlreplace hxlspec
hxlclean hxldedup hxlfill hxlmerge hxlselect hxltag
fititnt@bravo:~$ hxlhash https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3.tab
Traceback (most recent call last):
File "/home/fititnt/.local/bin/hxlhash", line 8, in <module>
sys.exit(hxlhash())
File "/home/fititnt/.local/lib/python3.8/site-packages/hxl/scripts.py", line 63, in hxlhash
run_script(hxlhash_main)
File "/home/fititnt/.local/lib/python3.8/site-packages/hxl/scripts.py", line 1010, in run_script
sys.exit(func(sys.argv[1:], STDIN, sys.stdout))
File "/home/fititnt/.local/lib/python3.8/site-packages/hxl/scripts.py", line 420, in hxlhash_main
with make_source(args, stdin) as source:
File "/home/fititnt/.local/lib/python3.8/site-packages/hxl/scripts.py", line 1103, in make_source
input = make_input(args, stdin)
File "/home/fititnt/.local/lib/python3.8/site-packages/hxl/scripts.py", line 1123, in make_input
return hxl.io.make_input(
File "/home/fititnt/.local/lib/python3.8/site-packages/hxl/io.py", line 388, in make_input
(input, mime_type, file_ext, specified_encoding) = open_url_or_file(
File "/home/fititnt/.local/lib/python3.8/site-packages/hxl/io.py", line 490, in open_url_or_file
content_type = response.headers['Content-type']
File "/usr/lib/python3/dist-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
KeyError: 'content-type'
fititnt@bravo:~$ curl -I https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3.tab
HTTP/2 200
date: Mon, 14 Dec 2020 01:58:13 GMT
content-length: 184662
set-cookie: __cfduid=d33feef07a6c9560027bd2a68bc0185271607911092; expires=Wed, 13-Jan-21 01:58:12 GMT; path=/; domain=.sil.org; HttpOnly; SameSite=Lax; Secure
x-content-type-options: nosniff
last-modified: Thu, 10 Sep 2020 21:09:00 GMT
cache-control: max-age=1209600
expires: Sat, 26 Dec 2020 07:40:25 GMT
x-request-id: v-4c238bc0-3c4d-11eb-a5a6-22000a9db97e
x-ah-environment: prod
age: 152268
via: varnish
x-cache: HIT
x-cache-hits: 43
accept-ranges: bytes
cf-cache-status: DYNAMIC
cf-request-id: 0700914ae50000f6f0b39e4000000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 6014518b0d9df6f0-GRU
fititnt@bravo:~$ curl -s https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3.tab | head
Id Part2B Part2T Part1 Scope Language_Type Ref_Name Comment
aaa I L Ghotuo
aab I L Alumu-Tesu
aac I L Ari
aad I L Amal
aae I L Arbëreshë Albanian
aaf I L Aranadan
aag I L Ambrak
aah I L Abu' Arapesh
aai I L Arifama-Miniafia
fititnt@bravo:~$ 
Ok. Vamos começar com o https://iso639-3.sil.org/, pois pode servir para fazer cruzamento com informações de outros lugares.
meta +key | #meta +value
-- | -- Metadata about | https://iso639-3.sil.org/code_tables/download_tables iso-639-3_20200515.tab | https://proxy.hxlstandard.org/data/edit?tagger-match-all=on&tagger-01-header=id&tagger-01-tag=%23vocab+%2Bid+%2Bv_iso6393_3letter&tagger-02-header=part2b&tagger-02-tag=%23vocab+%2Bcode+%2Bv_iso3692_3letter+%2Bz_bibliographic&tagger-03-header=part2t&tagger-03-tag=%23vocab+%2Bcode+%2Bv_3692_3letter+%2Bz_terminology&tagger-04-header=part1&tagger-04-tag=%23vocab+%2Bcode+%2Bv_6391&tagger-05-header=scope&tagger-05-tag=%23status&tagger-06-header=language_type&tagger-06-tag=%23vocab+%2Btype&tagger-07-header=ref_name&tagger-07-tag=%23description+%2Bname+%2Bi_en&tagger-08-header=comment&tagger-08-tag=%23description+%2Bcomment+%2Bi_en&url=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1mlc3zLkdgGRMOts36PiK2eFrMazgidKs%2Fview%3Fusp%3Dsharing&header-row=1&dest=data_view iso-639-3_Name_Index_202000515.tab | https://proxy.hxlstandard.org/data?tagger-match-all=on&tagger-01-header=id&tagger-01-tag=%23meta%2Bid&tagger-02-header=print_name&tagger-02-tag=%23meta%2Bname&tagger-03-header=inverted_name&tagger-03-tag=%23meta%2Bname%2Binverted&url=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1mczr4hXu5mi_Z5W-6kpHpD8qr7-UpzzY%2Fview%3Fusp%3Dsharing&header-row=1&dest=data_view iso-639-3-macrolanguages_20200515.tab | https://proxy.hxlstandard.org/data/edit?tagger-match-all=on&tagger-01-header=m_id&tagger-01-tag=%23meta%2Bid%2Bmacrolaguage&tagger-02-header=i_id&tagger-02-tag=%23meta%2Bid%2Blanguage&tagger-03-header=i_status&tagger-03-tag=%23status&url=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1lAKLg6jXOfiEaPcTlaeIf7ni6l7CO3mQ%2Fview%3Fusp%3Dsharing&header-row=1&dest=data_view iso-639-3_Retirements_20200515.tab | https://proxy.hxlstandard.org/data/edit?tagger-match-all=on&tagger-01-header=id&tagger-01-tag=%23vocab%2Bid%2Bv_iso6393_3lette&tagger-02-header=ref_name&tagger-02-tag=%23description%2Bname%2Bi_en&tagger-03-header=ret_reason&tagger-03-tag=%23status+%2Bretirement+%2Breason&tagger-04-header=change_to&tagger-04-tag=%23meta+%2Bz_change_to&tagger-05-header=ret_remedy&tagger-05-tag=%23description+%2Blong+%2Bi_en&tagger-06-header=effective&tagger-06-tag=%23meta%2Bretirement%2Bdate&url=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1IoDVsIzz_MVX_RpYYbARSdAH1qGfskX-%2Fview%3Fusp%3Dsharing&header-row=1&dest=data_view
Potencial para pegar os dados diretamente de https://iso639-3.sil.org/
No momento fizemos o rascunho de HXLate usando uma pasta no Google docs, porém reparei que o site oficial serve os arquivos com uma URL que parece ser fixa (o link para elas está na barra lateral de https://iso639-3.sil.org/code_tables/download_tables)
Neste caso, mesmo que o uso HXLate seja para gerar outros datasets, ainda poderia ser pertinente tanto para gente como para quem for trabalhar poder ter URL que possa processar diretamente esses datasets.
Vamos tentar nem mesmo precisar usar o Google drive. Com isso ficaria perfeito!!!