http://pantry.learningequality.org is offline for 24+ hours, preventing IIAB installs that need KA Lite's en.zip [and multilingual .torrent files to install content]

holta commented 4 years ago

@benjaoming is there someone at Learning Equality that we should ask about this?

The main file in question is:

http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip

Thanks @pumpkinpi85 for reporting this!

Tangentially related: #871

holta commented 4 years ago

@ivanistheone do you know if access to this file (en.zip) can somehow be restored?

Or might you know who we might talk to at Learning Equality, to get some kind of access to it?

(Thanks if possible !!)

holta commented 4 years ago

Just FYI, KA Lite's content torrent files in ~7 languages are also currently offline: https://ka-lite.readthedocs.io/en/latest/usermanual/userman_admin.html#downloading-videos-in-bulk

As these live here — on this same host/site (panty.learningequality.org) that appears offline for about 48 hours now: http://pantry.learningequality.org/downloads/ka-lite/0.17/content/

jvonau commented 4 years ago

I did an install last night into this morning had no issues.

Install KA Lite's mandatory 0.9 GB English Pack... (en.zip) --2020-09-22 05:52:15-- http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip Resolving pantry.learningequality.org (pantry.learningequality.org)... 104.18.9.53, 104.18.8.53, 2606:4700::6812:935, ... Connecting to pantry.learningequality.org (pantry.learningequality.org)|104.18.9.53|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 929916955 (887M) [application/zip] Saving to: 'en.zip' en.zip 100%[=========================>] 886.84M 4.40MB/s in 4m 5s
2020-09-22 05:56:21 (3.61 MB/s) - 'en.zip' saved [929916955/929916955]

holta commented 4 years ago

I did an install last night into this morning had no issues.

Ok, Strange! I've tried different ISP's over 3 days with no luck, i.e. personally I haven't been able to access http://pantry.learningequality.org Sunday, Monday and now Tuesday. Similar to @pumpkin85, who has a different ISP (in a different country).

As confirmed each day by https://downforeveryoneorjustme.com/pantry.learningequality.org

(Can anybody else reach http://pantry.learningequality.org today or recently, or know what's happening here?)

holta commented 4 years ago

This reports pantry.learningequality.org is offline for the 4th day in a row now: https://downforeveryoneorjustme.com/pantry.learningequality.org

All ideas appreciated!

jvonau commented 4 years ago

Don't you have a copy of the zip file somewhere? to upload to d.iiab.io?

holta commented 4 years ago

I don't have a copy of KA Lite's en.zip

i.e. http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip

holta commented 4 years ago

Noting that the torrent files (for approximately 7+ languages at http://pantry.learningequality.org/downloads/ka-lite/0.17/content/) are equally important.

So content can be installed.

jvonau commented 4 years ago

I have a copy in my pi-gen's cache, would you like a copy or wait this out?

jvonau commented 4 years ago

Um, maybe the site is just blocking viewing based on user-agent? I just tried wget by hand then cancelled out and the spew is:

pi@box:~ $ date Wed 23 Sep 15:46:21 BST 2020 pi@box:~ $ wget http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip --2020-09-23 15:46:38-- http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip Resolving pantry.learningequality.org (pantry.learningequality.org)... 104.18.9.53, 104.18.8.53, 2606:4700::6812:935, ... Connecting to pantry.learningequality.org (pantry.learningequality.org)|104.18.9.53|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 929916955 (887M) [application/zip] Saving to: ‘en.zip.1’ en.zip.1 0%[ ] 2.25M 1.87MB/s ^C

Looks to be working here...

holta commented 4 years ago

Um, maybe the site is just blocking viewing based on user-agent?

@jvonau have you tried with other browsers to confirm that's what's happening on your end?

Resolving pantry.learningequality.org (pantry.learningequality.org)... 104.18.9.53, 104.18.8.53, 2606:4700::6812:935, ... Connecting to pantry.learningequality.org (pantry.learningequality.org)|104.18.9.53|:80... connected.

FYI wget doesn't work here: (over Xfinity/Comcast and other major ISP's in the US)

pi@box:~ $ wget http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip
--2020-09-23 10:59:59--  http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip
Resolving pantry.learningequality.org (pantry.learningequality.org)... 104.18.8.53, 104.18.9.53, 2606:4700::6812:835, ...
Connecting to pantry.learningequality.org (pantry.learningequality.org)|104.18.8.53|:80... connected.
HTTP request sent, awaiting response...

Trying the IP address also fails:

pi@box:~ $ wget http://104.18.8.53/downloads/ka-lite/0.17/content/contentpacks/en.zip
--2020-09-23 11:01:09--  http://104.18.8.53/downloads/ka-lite/0.17/content/contentpacks/en.zip
Connecting to 104.18.8.53:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2020-09-23 11:01:09 ERROR 403: Forbidden.

FYI browsing to http://pantry.learningequality.org/downloads/ka-lite/0.17/content/ currently yields a 522 error Cloudfare page, with a link to https://support.cloudflare.com/hc/en-us/articles/115003011431-Error-522#522error

Whereas on Sunday (3 days ago) this yielded a similar page with a 524 error.

holta commented 4 years ago

I have a copy in my pi-gen's cache

Do protect that somewhere.

While the larger problem (working torrents, to install content) needs to be solved just as urgently so as get people back on their feet.

jvonau commented 4 years ago

Not having the zip file is blocking 'iiab' you could comment that stanza and the completed flag part out for now, remember that the stanzas are conditional on the install flags being absent. Once everything is back to normal, revert the change and the end user can just continue with 'iiab', the completed parts of are skipped and the install continues..

holta commented 4 years ago

FYI I've confirmed that KA Lite's (mandatory) English language pack cannot be downloaded/installed through (any) of the 3 traditional methods:

wget http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip
/usr/bin/kalite manage retrievecontentpack download en
Log in to KA Lite's Language Management / Update Languages page (http://box:8008/update/languages/) as user Admin > "Download or Update Language Packs" > "Select language pack" (pop-up menu no longer works, apparently as a result of pantry.learningequality.org being offline)

jvonau commented 4 years ago

For you... traceroute might help shed some light..

[jerry@box iiab]$ traceroute pantry.learningequality.org traceroute to pantry.learningequality.org (104.18.8.53), 30 hops max, 60 byte packets 1 hitronhub.home (192.168.0.1) 1.343 ms 3.079 ms 3.512 ms 2 * 3 rc3sc-be111-1.wp.shawcable.net (64.59.178.137) 20.437 ms 19.513 ms 21.271 ms 4 rc2nr-be26.wp.shawcable.net (66.163.73.146) 24.200 ms 24.810 ms 24.642 ms 5 rc3fs-be25.mt.shawcable.net (66.163.76.22) 48.014 ms 47.769 ms 51.836 ms 6 rc4fs-be5.mt.shawcable.net (66.163.66.58) 52.084 ms 35.227 ms 39.365 ms 7 198.32.181.56 (198.32.181.56) 39.512 ms 40.012 ms 40.112 ms 8 104.18.8.53 (104.18.8.53) 39.083 ms 37.920 ms 43.151 ms

holta commented 4 years ago

FYI traceroute pantry.learningequality.org and ping both work:

pi@box:~ $ ping pantry.learningequality.org PING pantry.learningequality.org (104.18.9.53) 56(84) bytes of data. 64 bytes from 104.18.9.53 (104.18.9.53): icmp_seq=1 ttl=59 time=11.4 ms 64 bytes from 104.18.9.53 (104.18.9.53): icmp_seq=2 ttl=59 time=17.5 ms 64 bytes from 104.18.9.53 (104.18.9.53): icmp_seq=3 ttl=59 time=11.8 ms 64 bytes from 104.18.9.53 (104.18.9.53): icmp_seq=4 ttl=59 time=12.3 ms ^C --- pantry.learningequality.org ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 7ms rtt min/avg/max/mdev = 11.392/13.255/17.526/2.489 ms

jvonau commented 4 years ago

Um, maybe the site is just blocking viewing based on user-agent?

@jvonau have you tried with other browsers to confirm that's what's happening on your end?

FF is blocked on my nuc, 'download failed' when prompted to save file after pasting the full url into the address box. I'm running pi-gen on the rpi4, wget works correctly from both machines.

FYI wget doesn't work here: (over Xfinity/Comcast and other major ISP's in the US)
pi@box:~ $ wget http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip
--2020-09-23 10:59:59--  http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip
Resolving pantry.learningequality.org (pantry.learningequality.org)... 104.18.8.53, 104.18.9.53, 2606:4700::6812:835, ...
Connecting to pantry.learningequality.org (pantry.learningequality.org)|104.18.8.53|:80... connected.
HTTP request sent, awaiting response...

maybe on a sh!t list to be rate limited?

Trying the IP address also fails:
pi@box:~ $ wget http://104.18.8.53/downloads/ka-lite/0.17/content/contentpacks/en.zip
--2020-09-23 11:01:09--  http://104.18.8.53/downloads/ka-lite/0.17/content/contentpacks/en.zip
Connecting to 104.18.8.53:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2020-09-23 11:01:09 ERROR 403: Forbidden.
I would expect that from any multi-site virtual_host based web server.

FYI browsing to http://pantry.learningequality.org/downloads/ka-lite/0.17/content/ currently yields a 522 error Cloudfare page, with a link to https://support.cloudflare.com/hc/en-us/articles/115003011431-Error-522#522error

Whereas on Sunday (3 days ago) this yielded a similar page with a 524 error.

Don't know what to say I can't reproduce the failure.

jvonau commented 4 years ago

Working 'ping' doesn't count for much. the traceroute is more important to help understand why my path to 104.18.8.53 works and yours does not.

ivanistheone commented 4 years ago

From Montreal, wget http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip returns ERROR 522 and same in browser

pinging @aronasorman in case he can check on this

holta commented 4 years ago

Is this some kind of web server issue? I tried this low-level HTTP/1.1 command on both of pantry's IPv4 IP addresses:

GET / HTTP/1.1
host: pantry.learningequality.org
<hit 'Enter' one more time>

Below are 2 resulting traces — oddly this result from 104.18.8.53 is different from 104.18.9.53 !

(Though in each case leading to a 522 error, about a minute after I enter the GET request.)

pi@box:~ $ telnet pantry.learningequality.org 80
Trying 104.18.8.53...
Connected to pantry.learningequality.org.
Escape character is '^]'.
GET / HTTP/1.1
host: pantry.learningequality.org

HTTP/1.1 522 Origin Connection Time-out
Date: Wed, 23 Sep 2020 16:30:41 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d0bf7bdb624cc09945bfeb823386fd96b1600878610; expires=Fri, 23-Oct-20 16:30:10 GMT; path=/; domain=.learningequality.org; HttpOnly; SameSite=Lax
Cache-Control: no-store, no-cache
cf-request-id: 055d6621a90000ff7ccc2d4200000001
Server: cloudflare
CF-RAY: 5d75a5cbaa36ff7c-BOS
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

c8
<html>
<head><title>522 Origin Connection Time-out</title></head>
<body bgcolor="white">
<center><h1>522 Origin Connection Time-out</h1></center>
<hr><center>cloudflare-nginx</center>
</body>
</html>

0

Connection closed by foreign host.

pi@box:~ $ telnet pantry.learningequality.org 80
Trying 104.18.9.53...
Connected to pantry.learningequality.org.
Escape character is '^]'.
GET / HTTP/1.1
host: pantry.learningequality.org

HTTP/1.1 522
Date: Wed, 23 Sep 2020 16:34:47 GMT
Content-Length: 0
Connection: keep-alive
Set-Cookie: __cfduid=d323c9ed4fadacc7d7f53bbcf6c50fd481600878794; expires=Fri, 23-Oct-20 16:33:14 GMT; path=/; domain=.learningequality.org; HttpOnly; SameSite=Lax
Cache-Control: no-store, no-cache
CF-Cache-Status: DYNAMIC
cf-request-id: 055d68efc90000ff88e3041200000001
Server: cloudflare
CF-RAY: 5d75aa900c07ff88-BOS
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

Connection closed by foreign host.

holta commented 4 years ago

to help understand why my path to 104.18.8.53 works and yours does not.

Clarification: I can connect to 104.18.8.53 and 104.18.9.53 e.g. over http (port 80) as well as traceroute and ping.

jvonau commented 4 years ago

pi@box:~ $ telnet pantry.learningequality.org 80 Trying 104.18.8.53... pi@box:~ $ telnet pantry.learningequality.org 80 Trying 104.18.9.53...

[jerry@box iiab]$ whois 104.18.9.53 snip NetRange: 104.16.0.0 - 104.31.255.255 CIDR: 104.16.0.0/12 NetName: CLOUDFLARENET

round-robin dns load-balancing on cloudflare's server farm.

jvonau commented 4 years ago

dig returns the same two ip mentioned above for both pantry.learningequality.org and learningequality.org with the latter being available. Opening pantry.learningequality.org just hangs FF. Think 'pantry' is proxyed to a different machine from the main domain.

just for reference:

wget https://learningequality.org/r/kolibri-deb-latest --2020-09-23 18:45:58-- https://learningequality.org/r/kolibri-deb-latest Resolving learningequality.org (learningequality.org)... 104.18.9.53, 104.18.8.53, 2606:4700::6812:835, ... Connecting to learningequality.org (learningequality.org)|104.18.9.53|:443... connected. HTTP request sent, awaiting response... 302 FOUND Location: https://storage.googleapis.com/le-releases/downloads/kolibri/v0.14.3/kolibri_0.14.3-0ubuntu1_all.deb [following] --2020-09-23 18:45:59-- https://storage.googleapis.com/le-releases/downloads/kolibri/v0.14.3/kolibri_0.14.3-0ubuntu1_all.deb Resolving storage.googleapis.com (storage.googleapis.com)... 216.58.192.208, 216.58.192.240, 216.58.192.176, ... Connecting to storage.googleapis.com (storage.googleapis.com)|216.58.192.208|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 33542432 (32M) [application/x-debian-package] Saving to: ‘kolibri-deb-latest’ kolibri-deb-latest 100%[==================================>] 31.99M 15.7MB/s in 2.0s
2020-09-23 18:46:01 (15.7 MB/s) - ‘kolibri-deb-latest’ saved [33542432/33542432]

aronasorman commented 4 years ago

Hi all, thank you for letting us know that this is down. We will work to bring it back up ASAP.

aronasorman commented 4 years ago

The pantry server should be up again! Thanks @ivanistheone for pinging us -- we'll add some alerting now on the en.zip URL :D.

holta commented 4 years ago

Profound thanks @aronasorman !

I can confirm the 7 essential .torrent language files are also now back online here: http://pantry.learningequality.org/downloads/ka-lite/0.17/content/

iiab / iiab

http://pantry.learningequality.org is offline for 24+ hours, preventing IIAB installs that need KA Lite's en.zip [and multilingual .torrent files to install content] #2538