Language pack download progress bar shows 100%

[bcipolli]

But it's just started downloading, and will continue for ~ 1 minute.

Perhaps this is due to http being forwarded to https? @jamalex any suggestions?

[jamalex]

Hmm... Is this new? I observed something similar previously, with it showing "1 of 4" and then jumping immediately to 100%, but I just assumed that was because stages 2 through 4 were... very fast!

[jamalex]

You can try http://keep.learningequality.org:8008/ as the central server (bypassing the proxy) if you want to see if it's an HTTPS issue.

[bcipolli]

Yes, this is new. The download stage has always been progressive; the others tend to jump because they're fast. I will try the above.

[jamalex]

Ok -- lemme know if there's anything I should try on the central server to help with the debugging.

[bcipolli]

OK. Tried with the above URL, same issue. Switched to the develop central server, works as expected, so I don't believe it's a client issue.

I will try updating code on the develop central server to make sure it's not a bug in the latest central server code.

[bcipolli]

Unfortunately, this is definitely a server config issue. As the server config guru, is this something you could debug @jamalex ?

(still worked with develop central server on the latest release-0.11 code)

[jamalex]

When you say "develop central server", do you mean the new staging server, or the old develop server?

[bcipolli]

kalite.learningequality.org:7007. I didn't call it adhoc as I didn't use that host name (to always test the end-user experience); I can call it adhoc in the future if that helps!

[jamalex]

Cool, will try testing now. Saying "adhoc" sounds good. Especially as we also have "kitchen", which is the Rackspace dev server. And the new staging server is "globe" (as in Shakespeare's Globe Theater... get it? :D)

[jamalex]

Hmm, so it seems to work for me, though it often downloads the langpack really quickly (but still shows the progress bar for a bit)...

[jamalex]

Polish worked smoothly, and showed 3 or 4 increases on the top progress bar before completing.

DEBUG:kalite:Threaded launch of command languagepackdownload with parameters (), {'lang_code': u'pl'})
DEBUG:kalite:Starting command languagepackdownload with parameters (), {'lang_code': u'pl'})
[22/Feb/2014 07:03:54] "POST /api/languagepacks/start HTTP/1.1" 200 17
INFO:kalite:Downloading language pack for lang_code=pl, software_version=0.11.1
DEBUG:kalite:Setting languagepackdownload to 5 total stages.
Downloading language pack 'pl'
INFO:kalite:Retrieving language pack: pl
[22/Feb/2014 07:03:54] "GET /api/updates/progress?process_name=languagepackdownload&_=1393081083176 HTTP/1.1" 200 267
[22/Feb/2014 07:03:54] "GET /api/updates/progress?process_id=5&_=1393081083177 HTTP/1.1" 200 267
[22/Feb/2014 07:03:56] "GET /api/updates/progress?process_id=5&_=1393081083178 HTTP/1.1" 200 284
[22/Feb/2014 07:03:58] "GET /api/updates/progress?process_id=5&_=1393081083179 HTTP/1.1" 200 270
[22/Feb/2014 07:04:00] "GET /api/updates/progress?process_id=5&_=1393081083180 HTTP/1.1" 200 270
Unpacking language pack 'pl'
INFO:kalite:Unpacking new translations
Creating static files for language pack 'pl'
Moving files to their appropriate local disk locations.
DEBUG:kalite:Moving dubbed video map to /home/jamalex/fle/ka-lite-clean2/data/i18n/dubbed_video_mappings.json
DEBUG:kalite:Removing emtpy directory
WARNING:kalite:Could not find downloaded exercises; skipping: /home/jamalex/fle/ka-lite-clean2/locale/pl/exercises
INFO:kalite:Moving 1460 subtitles from /home/jamalex/fle/ka-lite-clean2/locale/pl/subtitles to /home/jamalex/fle/ka-lite-clean2/kalite/static/srt/pl/subtitles
INFO:kalite:Removing empty source directory (/home/jamalex/fle/ka-lite-clean2/locale/pl/subtitles).
DEBUG:kalite:Moving /home/jamalex/fle/ka-lite-clean2/locale/pl/video_file_sizes.json to /home/jamalex/fle/ka-lite-clean2/data/content/video_file_sizes.json
Invalidate caches
DEBUG:kalite:Emptying cache i18n.SUPPORTED_LANGUAGE_MAP
DEBUG:kalite:Emptying cache i18n.DUBBED_VIDEO_MAP_RAW
DEBUG:kalite:Emptying cache i18n.DUBBED_VIDEO_MAP
DEBUG:kalite:Emptying cache i18n.YT2ID_MAP
DEBUG:kalite:Emptying cache i18n.ID2OKLANG_MAP
DEBUG:kalite:Emptying cache i18n.YT2LANG_MAP
DEBUG:kalite:Emptying cache i18n.CODE2LANG_MAP
DEBUG:kalite:Emptying cache i18n.LANG2CODE_MAP
DEBUG:kalite:Emptying cache i18n.LANG_NAMES_MAP
DEBUG:kalite:Emptying cache i18n.INSTALLED_LANGUAGES_CACHE
DEBUG:kalite:Emptying cache main.topic_tools.TOPICS
DEBUG:kalite:Emptying cache main.topic_tools.NODE_CACHE
DEBUG:kalite:Emptying cache main.topic_tools.KNOWLEDGEMAP_TOPICS
DEBUG:kalite:Emptying cache main.topic_tools.SLUG2ID_MAP
DEBUG:kalite:Emptying cache main.topic_tools.FLAT_TOPIC_TREE
INFO:kalite:Great success emptying the in-memory cache.
DEBUG:kalite:Clearing the web cache.
DEBUG:kalite:Great success emptying the web cache.
Finished processing language pack pl
DEBUG:kalite:Completing process languagepackdownload
[22/Feb/2014 07:04:02] "GET /api/updates/progress?process_id=5&_=1393081083181 HTTP/1.1" 200 267
DEBUG:kalite:Found language pack zh_CN
DEBUG:kalite:Found language pack en
DEBUG:kalite:Found language pack es
DEBUG:kalite:Found language pack el
DEBUG:kalite:Found language pack pt_BR
DEBUG:kalite:Found language pack pl
DEBUG:kalite:Found language pack bg
[22/Feb/2014 07:04:02] "GET /api/languagepacks/installed?_=1393081083182 HTTP/1.1" 200 3123

[jamalex]

Also, it's better to test against the staging server, rather than the old dev server (adhoc), since it's basically identical to the production config, and you have access to it too.

[jamalex]

In Firefox, when I clicked to download, it popped and then hid a message (django messaging system) and then showed the download bar briefly before hiding it again.

[jamalex]

Danish downloaded smoothly with visible progress bar movement in Firefox.

[bcipolli]

Sure thing. I tested against the adhoc server only to see if it might be a server configuration issue.

Glad to hear it may be local to my setup, will de-prioritize until some of the other larger issues are addressed. Thanks!!

[aronasorman]

Non-essential. Punting to 0.11.2.

[bcipolli]

Pulling back, as this can cause support requests / confusion that can be costly (time-wise) and turn off / confuse users.

[aronasorman]

@rtibbles do you have time to work on this?

[rtibbles]

Probably on Thursday.

[rtibbles]

Looking at this today, is this still extant, or was it just a local issue for @bcipolli ?

[rtibbles]

Still seems extant.

[rtibbles]

On smaller languages, the download always seems to be at 100%, but still takes a long time, and all other stages get jumped over faster than is visible to the user (in fact, apart from stage 2 - the download stage, only stage 6 even showed up in the responses to API calls - doubly odd, as the 'total number of stages' is only meant to be 5...).

[bcipolli]

Package sizes look good here: https://kalite.learningequality.org/api/i18n/language_packs/available

[rtibbles]

Cheers - tracing this morning.

On Fri, Mar 7, 2014 at 6:14 AM, Ben Cipollini notifications@github.comwrote:

Package sizes look good here: https://kalite.learningequality.org/api/i18n/language_packs/available

Reply to this email directly or view it on GitHubhttps://github.com/learningequality/ka-lite/issues/1589#issuecomment-37027167 .

Richard

[rtibbles]

Leaving until @jamalex has checked if this is an Apache config issue on the production central server.

[aronasorman]

@jamalex how's this going?

[jamalex]

I'll be getting back to this once I've submitted my final paperwork tomorrow; I deprioritized it as it isn't breaking anything badly, and is more of a UX issue.

If anyone else is looking into it in the meantime: Apache should be reporting file sizes just fine in the HTTP headers, so it's probably not that.

[bcipolli]

@jamalex Thanks for letting us know; in the future, a public update here on what you're thinking would be really helpful!

We've identified this as one of the core features needed to ship i18n. It's at the core codepath for the new feature, it's confusing to users whether the update is working (given the current broken UX), and these downloads can take a long time.

If you could update about when you think you'd be able to look at this today, that'd be great! We have to get this out the door on Wed. evening, and knowing your expected timelines will help us understand what else we might need to do to make that happen!

Thanks!

[jamalex]

Thanks for letting us know; in the future, a public update here on what you're thinking would be really helpful!

Sure thing -- I had talked to you directly about postponing this issue (and had gotten the impression it wasn't a blocker), but should have posted a note here too.

If you could update about when you think you'd be able to look at this today, that'd be great!

My final submission appointment with OGS is at 11am (frantically going through the final formatting now), and I should be out of there by noon, then a quick bite, then on it at 1pm.

[jamalex]

Currently blocked by #1714, but will start with a parallel fresh install soon so I can leave that to be resolved.

[rtibbles]

@jamalex If this is definitely not a config issue on the central server, I am happy to continue working on this - don't feel like you have to do it all! The API call is returning the correct filesizes, but when the download is initialized, the download process itself is not getting any filesize information (returning -1).

[bcipolli]

This issue is for the release-0.11.1 branch; #1714 is on the develop branch. Please work with @rtibbles to solve. Thanks!

[bcipolli]

Are these requests being redirected to https? If so, is it possible that this is due to the https redirect? Perhaps urllib2 has some funky issue? This was my only real idea over the past few days.

[jamalex]

ok, I'll test on a fresh release-0.11.1 install, and report back (haven't gotten far enough through to see what's going on yet)

[jamalex]

This was indeed an Apache configuration issue. Turns out the /static dir was being served by Apache itself, but the /media dir wasn't (it was falling through to Django, which was chunking the response). Fix is here: https://github.com/fle-internal/ansible-playbooks/commit/fb87b2b49d9af5fcaedcd4c9843ae69b620fad09

I've updated globe and tested this against there; upper progress bar is now smooth. Before I update keep, I wanted to check whether I should be putting it on the latest release-0.11.1, or leaving it on release-0.11. There seems to be new stuff on release-0.11 that hasn't been deployed; let me know when I should push the :red_circle:.

[bcipolli]

Release 0.11.1 to keep; 0.11 will become 0.11.2

[bcipolli]

Wow, I think that's the first time that my experience / knowledge gained at the big M$FT helped me! Chunked encoding FTW! :)

Thanks for tracking it down!

[jamalex]

Haha, I'm sure some of your other mad skillz were strengthened there too! :D

I've deployed to keep, and am testing. How exactly do I delete language packs now? The zip files don't seem to be kept around. Is it the locale directory I should empty...?

[bcipolli]

Easiest way is to check out release-0.11, where there's a button to delete. If you want to stick with release-0.11.1, delete the locale/[lang_code] directory.

[jamalex]

Ok, cool. After the fix in #1715, this now works on keep (i.e. with default settings: empty local_settings.py).

[jamalex]

I guess let's leave this open until #1715 is merged.

[rtibbles]

Phew - glad this was a configuration issue, because I had no damn clue what else it might be.

[bcipolli]

1715 has been addressed in the short-term, this is fixed.. closing!