New mappings need to be integrated

[tschleider]

After @mpuren and Pierre reworked some / all of the existing mappings I still need to implement some of the changes.

[rtroncy]

This seems a very general issue. Is there a way to see the exhaustive list of changes that have been made? We can of course look at the history of each google document but this is not very convenient! Ideally, I would suggest to list of museum source with a checkbox in this github and to describe the changes that need to be dealt with so we can check the boxes upon commits.

[tschleider]

@mpuren said she marked every change in the paths as red.

[rtroncy]

The phrasing of this issue is still too general in the sense that we don't see immediately what needs to be changed and for which museum.

@tschleider Go through each museum mapping file, and report in this issue when you have validated all changes colored in red in the google docs. Do close the comments in google docs and change the colors as well.

[tschleider]

I went through everything once again and there are certain general topics that need to be addressed (next to some smaller ones):

• E8: I once deactivated most of the Acquisition mappings, originally temporarily, because they were using P2 in an undesired way (see #35 ). I need to add them again, eventually with another property instead.
• E30: All mappings concerning E30 (Right) are either missing or not working.
• E7: There seems to be a bug with E7 (Activity), too, that needs to be fixed. Those mappings are not in the KG.
• E9: I also deactivated E9 (Move) systematically, because the mapping was replaced with a new class that we can still not use. I asked for the mapping to return to E9 in the meantime, but now it needs to be implemented again.
• There are a few issues concerning the crawler / the JSON format.
• Some discussions in the mapping docs are still not solved, I went through all of them and am optimistic that we can close them soon.

Full status: https://docs.google.com/spreadsheets/d/1xcugWoTENsk-g7QC4fz4AeLn-VbJNgvZbwxXDTIMmr8/edit?usp=sharing

[tschleider]

• E8: Reactivated acquisition, where it works
• E30: Fixed rights and added it to several converters where it was missing (see also #60 )
• E7: Bug fixed
• E9: Reintroduced where it makes sense.
• We still have an ongoing email discussion about the fields "Embroidery", "Galloon" and "Lining" that only concern UNIPA.

Except for that we need to do some crawler updates, which I just communicated to @ehrhart :

• One important field for VAM is not crawled ("Subject depicted")
• The fields "Bibliography" and "Expositions" for Imatex are not crawled in the usual way and make conversion complicated.

[rtroncy]

New crawling has been done while ago. What is the status now?

[ehrhart]

IMATEX has been recrawled a while ago.

VAM still has to be updated (https://github.com/silknow/crawler/issues/36) and re-crawled. I'll keep you posted.

[ehrhart]

After verification the crawler for VAM has already been updated and the dump from July contains the "Subject depicted". The depicted subject is split into multiple fields on VAM's API, based on the type of subject. See https://github.com/silknow/crawler/issues/36 for the list of fields.

[tschleider]

Thanks @ehrhart ! I'll follow up by finishing the converter and will close the issue afterwards

[tschleider]

Ok I updated the converter and it can handle the new format now (which concludes this issue itself).

However, @ehrhart , it seems that the newer dump is way smaller than the older dumps on ownCloud. I therefore mixed them and kept the converter compatible to both. Is there a reason that the newer dump is much smaller than before?

EDIT: Just in light of @ehrhart 's new input, here the number of records in the VAM dump from December 2020 (most recent one up until now): 7747 records.

[ehrhart]

@tschleider I updated VAM crawler with an additional fix, but I just wanted to note that the number of results is still lower than what we used to have previously. However, I don't know if it was due to incorrect crawling in the past, or that VAM changed something to their dataset/API, or both.

I am crawling using 3 methods as described by the logbook, for a total of 943 results (some methods return similar results which is why we only have 943 results instead of 1155):

• (487 results) dates: 1400-1900, collection: Textile and Fashion Collection, technique: weaving, material: silk (textile)
• (35 results) keywords: silk, material: paper, collection: Textiles and Fashion Collection, dates: 1400-1900
• (633 results) keywords: textile design, material: silk (textile)

I will share the new dumps soon if that sounds okay.

[ehrhart]

VAM Records: vam_records_20211110_5.tar.gz VAM Files: vam_files_20211110_5.tar.gz

[tschleider]

Thanks for the new dump and the explanation, @ehrhart . It makes sens how you download the objects, but this is still a huge reduction of data from the KG (if I would simply replace the old dump of ca. 7700 files with the new one of less than 1000 file it's reduction of almost 7000 files).

What do you think we should do @rtroncy ?

[rtroncy]

I would keep the oldest VAM dump. This issue is now old and I lost track. Why did we want to re-crawl it in the first place? @tschleider Can you summarize the current issue?

[tschleider]

@rtroncy Their website got updated and they added a field for "Subject depicted", that would be of course highly relevant as we don't have a lot of (English) data for subject depiction. But now the new harvest (@ehrhart explains how he does it above) yields much less objects.

I would also rather keep the old dump as of now, but I could "mix" the files (so if a object has a newer download including "subject depicted" it will be used instead of its old version, otherwise it's the old dump).

What do you think?

[rtroncy]

A smart merge of the 2 dumps would be idea so a BIG +1 but I don't want that this takes too much time. You're the best to judge.

[tschleider]

I combined the most recent dump with the last one that was big and re-converted VAM (it's online). I also uploaded this combined dump to OwnCloud: https://silknow.uv.es/owncloud/index.php/apps/files/?dir=/Data&fileid=626898

Everything discussed in this issue has been integrated