Closed kelson42 closed 3 years ago
Seems to be the same error as #112 . It has already been fixed in the 2.1.8.dev0 version. I see that the container that the recipe uses is 2.1.7 . I'm running it on dev to check once if the fix works for this too.
We should probably release a new version then. This fix is already 2 monthes old. Any chance to get that today so we can relaunch the recipe?
We should probably release a new version then. This fix is already 2 monthes old. Any chance to get that today so we can relaunch the recipe?
I launched on the dev container. And it passed - https://farm.openzim.org/pipeline/5f61b5bb65c499a28faedd75 I checked the logs and they seem good. Haven't tested the ZIM though as I think its still in the quarantine.
@satyamtg I have to reopen the ticket as the bug seems not (fully) fixed, see https://farm.openzim.org/pipeline/d683223bedd943b94fb67ff5/debug with khan-videos_bn_playlists recipe.
@satyamtg I have to reopen the ticket as the bug seems not (fully) fixed, see https://farm.openzim.org/pipeline/d683223bedd943b94fb67ff5/debug with khan-videos_bn_playlists recipe.
I think we need to extend the language mapping to handle as this seems the same error as earlier. Running a test to find which language code breaks it. Will also add a better failure message (one that displays the language code that failed)
Might also be interesting to check that earlier in scraper as this happens and fails post-download
Same for teded_en_all, see https://farm.openzim.org/pipeline/4548223bedd943b91f808ff5/debug
@kelson42, youtube sometimes uses non-standard codes for subtitles. That's something we have to react to when we encounter it but it's not a regression.
I've added mapping for the khan-videos-bn
culprit (zh-Hans-CN
) and I've tried to find the list of all possible codes from the YT subtitle uploads UI. I found a couple of codes that would have failed and mapped those as well.
Unfortunately, that UI had no reference of that zh-Hans-CN
code for instance so it means the UI doesn't expose all the codes they are using, meaning we might encounter a similar issue in the future.
I've added a debug print of such code so that in the event of a similar issue, we'll save a lot of time and be able to fix it immediately.
So, next time, open a new ticket with the lang code.
And you mentioned khan-videos_es_playlists
but this one has noting to do with this. it's a recipe issue.
As for teded, it was zh-Hant-TW
.
Problem is still there, see https://farm.openzim.org/pipeline/825a5abd771312617d4cc506/debug:
[info] Writing video subtitles to: /output/tmpm2lbegcu/videos/zm3TXDZrifU/video.bg.vtt
[info] Writing video subtitles to: /output/tmpm2lbegcu/videos/zm3TXDZrifU/video.en.vtt
[info] Writing video subtitles to: /output/tmpm2lbegcu/videos/zm3TXDZrifU/video.ka.vtt
[info] Writing video subtitles to: /output/tmpm2lbegcu/videos/zm3TXDZrifU/video.ko.vtt
[info] Writing video subtitles to: /output/tmpm2lbegcu/videos/zm3TXDZrifU/video.sw.vtt
[info] Writing video subtitles to: /output/tmpm2lbegcu/videos/zm3TXDZrifU/video.ta.vtt
[info] Writing video subtitles to: /output/tmpm2lbegcu/videos/zm3TXDZrifU/video.vi.vtt
[youtube2zim::2021-03-28 07:43:02,086] INFO:retrieve channel-info for all videos (author details)
[youtube2zim::2021-03-28 07:43:02,086] DEBUG:query youtube-api for Video details of 31 videos
[youtube2zim::2021-03-28 07:43:02,186] INFO:download all author's profile pictures
[youtube2zim::2021-03-28 07:43:02,186] DEBUG:query youtube-api for Channel #UC4a-Gbdw7vOaccHmFo40b9g
[youtube2zim::2021-03-28 07:43:02,382] INFO:update general metadata
[youtube2zim::2021-03-28 07:43:02,504] INFO:creating HTML files
[youtube2zim::2021-03-28 07:43:02,880] ERROR:Failed to get language details for zh-Hant-HK
[youtube2zim::2021-03-28 07:43:02,881] ERROR:FAILED. An error occurred: Not a valid iso language name/code
[youtube2zim::2021-03-28 07:43:02,881] ERROR:Not a valid iso language name/code
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.13-py3.8.egg/youtube2zim/entrypoint.py", line 202, in main
scraper.run()
File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.13-py3.8.egg/youtube2zim/scraper.py", line 313, in run
self.make_html_files(succeeded)
File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.13-py3.8.egg/youtube2zim/scraper.py", line 861, in make_html_files
subtitles = get_subtitles(video_id)
File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.13-py3.8.egg/youtube2zim/scraper.py", line 839, in get_subtitles
return sorted(map(to_jinja_subtitle, languages), key=lambda x: x["name"])
File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.13-py3.8.egg/youtube2zim/scraper.py", line 827, in to_jinja_subtitle
subtitle = get_language_details(YOUTUBE_LANG_MAP.get(lang, lang))
File "/usr/local/lib/python3.8/site-packages/zimscraperlib/i18n.py", line 161, in get_language_details
raise exc
File "/usr/local/lib/python3.8/site-packages/zimscraperlib/i18n.py", line 157, in get_language_details
lang_data, macro_data = get_iso_lang_data(adjusted_query)
File "/usr/local/lib/python3.8/site-packages/zimscraperlib/i18n.py", line 77, in get_iso_lang_data
raise NotFound("Not a valid iso language name/code")
zimscraperlib.i18n.NotFound: Not a valid iso language name/code
@rgaudin Why the error message seems to still do not report the iso code which has a problem?
I suppose this recipe is not using the latest version. what version is it ?
2.1.13, it is using the latest version, see https://farm.openzim.org/recipes/khan-videos_en_playlists
The language code is in the log you supplied:
[youtube2zim::2021-03-28 07:43:02,880] ERROR:Failed to get language details for zh-Hant-HK
Fixed. Please retry with :dev
in a moment
Still a problem: ERROR:Failed to get language details for zh-Hans-SG
https://farm.openzim.org/pipeline/aa245a517a9c06751eaad606/debug (khan-videos_en_playlists)
Fixed. Please open a new ticket next time and use a different wording: those are new languages that we need to map and we don't have a list of those. Alternatively, we could also not fail and just display the language code in the subtitles select. Open a ticket for that if you think that's appropriate.
https://farm.openzim.org/pipeline/5f5f3154cf078db7bac5284e/debug