search

Signbank / Global-signbank

An online sign dictionary and sign database management system for research purposes. Developed originally by Steve Cassidy/ This repo is a fork for the Dutch version, previously called 'NGT-Signbank'.
http://signbank.cls.ru.nl
BSD 3-Clause "New" or "Revised" License
19 stars 12 forks source link

Signbank API: When updating senses, the update is not reflected in last_timestamp #1239

Open rem0g opened 2 months ago

rem0g commented 2 months ago

We are running updater every 5 minutes and based on since_timestamp -5 minutes we retrieve a json file. And i noticed there is no change reflected in the json file when we update senses on signbank, other changes like phonology are fine.

susanodd commented 2 months ago

This is probably because senses are comprised of many relation objects and not a gloss field.

This only happens when you have only updated the Senses and no other fields, right?

Probably the query on Gloss objects needs to be unioned with GlossSense objects.

Does it work correctly if you set the time interval to something longer than 5 minutes? (There could be a lag in Django updating things, since it evaluates some things lazily. Or it needs some fetch-related commands in the queries.)

@Jetske can you think about this? You are deleting the old senses and making new ones when updating, is that correct?

susanodd commented 2 months ago

I looked at some of the recently created glosses on NGT. I see there are some numbers that were created and then the senses were updated because the umlaut was some weird characters. Are these the glosses you are trying to retrieve that were updated? The update shows up in the Revision History of Gloss View.

https://signbank.cls.ru.nl/dictionary/gloss/48084/history I'll see if the last_timestamp can search in the revision history data. The difference is only 2 minutes. For this one, it's less than 1 minute between creation and update: https://signbank.cls.ru.nl/dictionary/gloss/48085/history Can you give an example of a gloss that didn't show up as being updated, if this isn't one?

rem0g commented 2 months ago

The change should be reflected as we look to timestamp now - 5 minutes, so when i change the sense the glos doesnt show up in the glosses JSON file. When phonological information is updated it does.

I have tested with this one: https://signbank.cls.ru.nl/dictionary/gloss/47350/. I do see senses update in the query but nothing in JSON.

susanodd commented 2 months ago

Okay, thanks! I will try to make it show up using the approaches mentioned above.

The query that looks at the timestamp uses the Gloss field lastUpdated.

Senses do not have this field. It is updated whenever a gloss object is saved. I guess we could artificially save a gloss to force it to be updated.

susanodd commented 2 months ago

I'm looking at the code where things are saved. But the code is already doing a gloss.save after it updates the senses. So it's very weird that it is not registering as being updated. (It's already doing a dummy save on the gloss when the senses are updated. Perhaps because the gloss itself is not being changed -- only the relations with senses -- Django knows it's not been updated so doesn't update the lastUpdated field (?)

I revised this to update the field explicitly.

(But updating the senses does not work for me at all. There's a different bug saving the senses to the revision history. It looks like the str does not work correctly on the dict values. @Jetske will need to check that.)

susanodd commented 2 months ago

This has been deployed.

As far as I can tell, this works now.