feature: metadata extraction - [merged]

[appledora]

Merges 41-metadata-extraction -> main

Extracts all the metadata from the dump json, except for ["article_body", "url", "namespace", "name", "in_language"] keys.

Closes #41

[appledora]

requested review from @martingerlach

[appledora]

added 4 commits

• d7a7f0e9...1dd9f0ec - 2 commits from branch main
• e6a3551a - update: dump.py remove irrelevant customization
• 5fb1f874 - update: html string in context.py replaced with json object

Compare with previous version

[appledora]

added 1 commit

• eee9c11b - update: html string in context.py replaced with json object

Compare with previous version

[appledora]

added 1 commit

• 602c5bce - update: html string in context.py replaced with json object

Compare with previous version

[appledora]

added 1 commit

• c0eca09e - update: page_namespace id added

Compare with previous version

[appledora]

In GitLab by @geohci on Aug 23, 2022, 21:09

Commented on src/parse/article.py line 26

I do think we'll want to remove this attribute. Right now I think it's just misleading ('en' is the language, which we already have, not the page_namespace). What's the reasonining for keeping this in? also, the page_namespace_id implementation looks good to me -- thanks!

[appledora]

In GitLab by @geohci on Aug 23, 2022, 21:10

Commented on src/parse/utils.py line 248

let's add a quick comment explaining why we skip these

[appledora]

Well, this attribute is actually used in the wiki link namespace extraction task. The NAMESPACE dictionary is nested. The first key is a namespace(I used to think it's only language acronyms, but it also contains simple) , inside this are the actual wiki namespaces like article and talks. To get to the actual namespace we have to do something like NAMESPACE[primary_namespace][secondary_namespace]. The primary namespace for the wikilinks come from the page_namespace. I actually should call it something other than page_namespace. SUggestions?

[appledora]

added 1 commit

• 31a4370b - Update utils.py

Compare with previous version

[appledora]

donezo

[appledora]

I just checked and found that the page_namespace and the page_namespace_id do not correspond to each other, so we DEFINITELY have to change it. Good grief =_=

[appledora]

changed this line in version 8 of the diff

[appledora]

added 2 commits

• 41e2fdbf - update: page_namespace renamed to primary_namespace
• d545fc86 - Merge branch '41-metadata-extraction' of...

Compare with previous version

[appledora]

In GitLab by @geohci on Aug 23, 2022, 21:27

Commented on src/parse/article.py line 26

ahhh I think I understand now. I had forgotten about our prior conversation about this (https://gitlab.wikimedia.org/repos/research/html-dumps/-/merge_requests/12#note_9977). sorry, let me try to clarify:

• what you are calling secondary_namespace is what I mean by namespace. This can have two forms -- the numeric identifier (0) or the prefix/name (Main). I prefer that we record the numeric identifier, which is what you have under page_namespace_id. no change needed there.
• what you are calling primary_namespace is the database name for the wiki. This is what you are currently extracting as self.page_namespace. You don't need to change the extraction but to avoid confusion, you actually want to call this self.wiki_db to match how it's usually referred to (i earlier suggested self.wiki but self.wiki_db is even clearer).
• we also have self.language, which is not relevant for namespaces but we should continue to extract. It's usually the same as self.wiki_db but there are some crucial differences -- e.g., simple is the wiki_db but en is the language for Simple English Wikipedia. again, no changes needed there.

[appledora]

added 1 commit

• 4ee97b6b - update: primary_namespace renamed to wiki_db

Compare with previous version

[appledora]

made the renaming changes!

[appledora]

In GitLab by @geohci on Aug 23, 2022, 21:58

Commented on src/parse/article.py line 26

perfect, thanks!

[appledora]

In GitLab by @geohci on Aug 23, 2022, 21:58

resolved all threads

[appledora]

In GitLab by @geohci on Aug 24, 2022, 24:57

mentioned in commit 03cb911b3de5edd6c1a53784deb9db29940a5d49