TS Background Tasks - Showing extracts created for wrong workbook with same name in different project

[wachhoa]

Hi @mcoles ! I noticed in all the versions of TS Background tasks that if there's a workbook with the same name in two different projects (that have different names) it doesn't differentiate which workbook is actually refreshing. It shows the workbooks in both the projects refreshing, and it's incorrect. I think it may be because background jobs table is joined to workbooks and data sources tables by name. There's no workbook ID or data source ID in background jobs tables. The one field I noticed that has "possibly" the workbook/data source ID in it in background jobs table is "args". It has a bunch of other stuff in the field though. Do you have any suggestions?

[mcoles]

Hi @wachhoa. Yeah, this is a known issue because you can't always get a perfect join given the vagaries of the data structures. If extracts are running normally with a scheduled task assigned to them, it works fine...but if that task is then deleted, the structure is no longer there and the association is lost. So then the code falls back to name and type, which is okay in some circumstances, but certainly not all.

Args is a field I've been reticent to use both because parsing data out to use in joins hurts performance, but also because it's an unstructured text field that there is far less rigor around on the development side. I did have to resort to using it for Data Alert IDs, which you can see in the Custom SQL, but for other things I've been avoiding it. I'll do some playing with this now that I've revisiting it...since we're advising this data be extracted and incrementally refreshed now anyway, it's probably a better approach.

[wachhoa]

Thanks @mcoles for your response. When you say "since we're advising this data be extracted and incrementally refreshed now anyway, it's probably a better approach", does that mean incremental refreshes would solve the issue? We are doing that by the way. Thanks for looking into this!

[mcoles]

Sorry, what I mean is that the performance issue isn't relevant if you're extracting, so I may as well bite the bullet and implement the fix you suggested. I did that yesterday and am working on some other changes, hope to deploy the new versions soon.

[wachhoa]

Oh okay, nice! I tried doing this in Postgres and couldn't figure out what type of parse I would need to make it work, since args doesn't seem to have a standard format.

[mcoles]

This is resolved (as much as possible, anyway) with the 04.01 release.

[wachhoa]

Thanks @mcoles ! So have to wait until 2023.1, correct?

[mcoles]

Yes, although you could manually grab the changes from the Custom SQL in 04.01 and retroactively apply it to 03.01, and just leave the changes due to the new projects_contents table joins if you didn't want to wait.