Health-Informatics-UoN / Carrot-Mapper

Carrot: Convenient And Reusable Rapid Omop Transformer.
https://carrot4omop.ac.uk
MIT License
12 stars 3 forks source link

Processing and importing concept_ids for bigger tables is taking a long time #754

Closed erummas closed 2 weeks ago

erummas commented 2 weeks ago

Is there an existing issue for this?

Current Behavior

Hello, currently for bigger tables with large number of field values I am trying to import concepts_ids for the codes that did not map automatically, however the process is very time consuming as it takes minimum 15 seconds and whilst it's processing I cannot add concept_id for another field so have to wait till it's done. I have not noticed this behaviour in smaller tables of the same scan report and it takes 1-2 seconds, so fairly quicker.

Expected Behavior

No response

Steps To Reproduce

Home /

Scan Reports /

SHARE_noGP /

share_smr01_conditions.csv /

CONDITIONICD10

Environment

- OS:
- Other environment details:

I'm part of a Project Team

No response

Anything else?

No response

Are you willing to contribute to resolve this issue?

None

AndyRae commented 2 weeks ago

Hi @erummas - I can't access that scan report, but this is unsurprising and related to the ability of the tool to handle large tables currently.

As we've been working directly on this, could you check the test instance for whether it works better, for what I think is a very similar Scan Report (it fits the above structure): https://ccom-test.azurewebsites.net/scanreports/232/tables/886/fields/15850/

We're intending to deploy this to production this week if it passes testing.

erummas commented 2 weeks ago

Thanks @AndyRae it's working great on test. Very quick. Can't wait for it to be deployed to prod. Thanks again

AndyRae commented 2 weeks ago

That's great to hear @erummas - we'll aim for a production release tomorrow morning, but leave this issue open until then.

Testament to @AndrewThien + @Dev-Akashili for their work on this.

(fixed in #722 )

erummas commented 2 weeks ago

Thanks @AndrewThien @Dev-Akashili brilliant work.

Dev-Akashili commented 2 weeks ago

Thanks! @erummas

Thanks for the support! @AndyRae

AndyRae commented 2 weeks ago

We've now deployed this fix.

erummas commented 2 weeks ago

Thank you @AndyRae, I can see the difference already. However, there is an unusual behaviour, whenever I add a concept, after updating that record vanishes from the page. I did check the next page to see if it moves there but it doesn't show.

erummas commented 2 weeks ago

The above doesn't happen when I sort the values in an order

AndrewThien commented 2 weeks ago

Thank you @AndyRae, I can see the difference already. However, there is an unusual behaviour, whenever I add a concept, after updating that record vanishes from the page. I did check the next page to see if it moves there but it doesn't show.

Thanks for reporting. But sorry, I don't fully understand "that record vanishes from the page" mean. Can you maybe take a screenshot or short video?

erummas commented 2 weeks ago
mapper-1 mapper-2 mapper-3

Mapping code R54.X : You can see from the Search bar the code after mapping is not on the page.

AndyRae commented 2 weeks ago

Going to just keep this open until we've deployed the fix to production.