DCT - strange mistake, reason unclear

[shariboodts]

We've noticed problems in the DCT (case in point: https://passim.rich.ru.nl/dct/details/82/ with Montecassino 12 as pivot manuscript):

First problem: PS-AU s 249 (item 76 in pivot manuscript) should be visible in DCT as item 70 in Vat. lat. 3836. The AF is properly linked to manifestations in both manuscripts. Why does it not show up in the DCT? Iris:

Second problem:

If this is indeed the case, that is not how it should work, DCT's should always use the most recent, updated version of the manuscripts/AF's the include.

Sometimes when clicking 'View all' on Rows drop-down menu, there is an error: Is this related to the problems described above?

[Irisdenis]

Edit: Regarding the first problem: this issue seems to be related to the same bug mentioned in issue #745. When first opening the DCT in question, PS-AU s 249 (nr 76) is not shown, even though it is a match. Only after clicking the option 'Rows = show all', PS-AU s 249 (nr 76) appears as a match in the table. When re-clicking 'Rows = Matches only', the displayed matches seem correct (i.e. nr 76 is displayed). Ergo, the problem is connected to the displaying of matches (not the recognition of matches), just like the examples mentioned in issue #745

[Irisdenis]

Another issue, possibly related to problem 2, is problem 3:

• In the DCT https://passim.rich.ru.nl/dct/details/23/, with Montpellier H 59 as pivot manuscript, the 'authors' displayed for Montpellier H 59 are not the same ones as the 'attributed authors' defined in the manuscript (e.g. we still find 'Anonymus' as author here, whereas in the manuscript details view, no author is defined. This should be displayed as three dots (...) in the DCT).
• When deleting and re-adding the manuscript to the DCT, the problem persist.
• When creating a new DCT with this manuscript, the manuscript's authors are displayed correctly.

So: it seems as if old DCT's like this one consistently present the information as it was present in the manuscripts at the time they were added to the DCT (i.e. they are stable, unchanged), even after deleting and re-adding the manuscript to the DCT. Question: Is this indeed where the problem originates?

This results in an incorrect view of the manuscripts in the DCT (i.e. the ms as in the DCT does not correspond to the ms details page). Ideally, a DCT should automatically retrieve 'changed/corrected' data from the manuscripts, or users should at least be able to delete and reload manuscripts that were since corrected (see above).

[ErwinKomen]

Problem 1

This problem was treated in issue #745, and has been resolved. For the most part, this was not a problem, but the use of hidden rows by the researcher. I've added information: when a DCT has hidden rows, then a clear Warning is added. And hovering over that warning shows the simple method to remove the hidden rows.

[ErwinKomen]

Problem 2

One large question here is whether the DCT is generated from AF data on a given moment, and not updated, even if that AF is updated.

1. A DCT is based on a research set, which is shown by /rset/details/NN/
2. The research set consists of set lists that can be manuscripts, SSG (private) datasets or historical collections
3. The 'contents' of a research set are indeed stored as (static) JSON inside the field contents of that research set, in order to facilitate quick responses
4. There is a link between the research set (with static data) and each of the research set contents (Manuscript, SSG dataset, Historical Collection): that link is via the Set List
5. There is a re-calculate function (re-calculation of the research set's contents), and it is ResearchSet / update_ssglists(). It is triggered in these situations:
   1. When a SetList is added to the DCT
   2. When the Pivot changes
   3. When any changes are made in the DCT by the user (name, order, hidden rows etc)
6. So, in terms of updating, whenever a Manuscript or a Collection is 'changed', re-calculation of the research set's contents could take place. This means:
   1. Whenever the contents of a Manuscript or Collection changes (i.e. addition, deletion):
      1. Re-calculate the SetList contents field to which the Collection or Manuscript is connected
   2. Whenever a SetList item is re-calculated, automatically re-calculate the ResearchSet item
7. HOWEVER, is this what one wants?
   1. When one has made a DCT, is at a conference to show it, and its ingredients have changed, the previous contents would automatically change - that may be undesirable
   2. Perhaps better would be:
      1. When there is indication that the contents could be changed, because a connected Collection or Manuscript has changed, then do:
         1. Change the status of the SetList item to stale
         2. Change the status of the ResearchSet to stale
         3. Add a comment to the user "A related manuscript, dataset or historical collection has changed" with a button Update DCT to execute an overal update
      2. Or even more simple: just add a button Update all in order to re-calculate and update research set contents (i.e. all setlists as well as the ResearchSet contents)

Implementation

Q: What are the triggers for the 'contents' of a manuscrip or collection changing?

1. Manuscript contents change triggers
   1. Add SermonDescr to manuscript => mostly SermonDescr create
      1. Add code in the view's after_new() method - no, wrong place, removed
      2. Added code in the model's save() method, triggered by the creation of the item - done
   2. Remove SermonDescr from manuscript => mostly SermonDescr delete
      1. Add code in the model's delete() method - okay
   3. A change happens to the related AFs of a SermonDescr in a manuscript
      1. This involves the model SermonDescrEqual, and then its parameter manu
      2. Add code to the model's delete() method - done
      3. Add code to the model's save() method, picking out where an item is created - done
2. Collection contents change triggers
   1. Add item to collection
      1. Added code to the save() method of model CollectionSuper that is triggered when an item is created
   2. Remove item from collection
      1. Added code to the delete() method of model CollectionSuper

[ErwinKomen]

Problem 3

Problem:

Deleting and re-adding the manuscript to a DCT does not update its contents correctly

This should now not be necessary anymore. What I do need to do here: a one-time (adaptations.py) update of all setlists, so that they are 'fresh' again

1. Starting from the collections: adapt_coll_setlists()
2. Starting from the manuscripts: adapt_manu_setlists()

Double checking

Looking at DCT 23, research set 4, manuscript Montpellier H 59, setlist id 30, sermon linked to PS-AU s Gue 1. There seemingly is an author mismatch:

1. Manuscript shows author Augustinus Hipponensis
2. DCT shows author Anonymus

But all is well:

1. Manuscript shows the sermon author, i.e. the attributed author
2. DCT shows the authority file author in the first column popup, i.e. what has actually been established for this in the AF PASSIM 051.0040, which is Anonymus
3. DCT shows the correct sermon author (attributed author) in the column author

Result: all seems to be well now.