search

ArctosDB / arctos

Arctos is a museum collections management system
https://arctos.database.museum
61 stars 13 forks source link

Request - Async Parts Table #8326

Open acdoll opened 3 days ago

acdoll commented 3 days ago

Would it be possible to add an async request option to the Part View and Download page? (https://arctos.database.museum/Reports/part_data_download.cfm?table_name=api_acdoll_20241121101143784_064)

I got what I needed by pulling a search with JSON Parts and getting an async from there, but I'll have to format it myself. It'd be great to have it divided out as you have it in the view table, but large datasets require to much memory to display so you can't get to the download.

image

dustymc commented 3 days ago

Need more info - how'd you get there?

acdoll commented 3 days ago

I am trying to generate a reference list of partIDs to resolve some tissue issues we're dealing with. I pulled a search for our whole mammal collection (https://arctos.database.museum/search.cfm?guid_prefix=DMNS%3AMamm) and then went to the View/Download Part in the Tools dropdown. I'm wondering if we can add the async request to this page before selecting one of these options (or, probably a request for each option): image It does load and I can actually get all of the partID's if I use the 'Bulk Edit Part View' option, but I can't tell which of these are subsamples of other parts (I'd like to exclude all of the subsamples from my final list). The other two options seem to be pulling too much data so that my browser (or computer's RAM) can't handle it trying to display all that info. I just want to download, I don't need to view it in the browser. I'm sure I could get what I need through some writeSQL work, but I thought a async request would be handy here since you've already done the work of selecting/pulling the data I'm interested in.
As I said, I think I was able to get what I need out of the main search JSON Parts, but that has taken a fair amount of OpenRefine/Excel work after the download. I'm certainly open to other ideas.

dustymc commented 3 days ago

Thanks.

First glance, this looks like a job for SQL - seems to be some rare-sih need, not easily done in the UI, and maybe simple/quick with my tools - just let me know what you need (but I think it's also often not that simple, do you really need a flattened view, can you deal with normalized - in which case I can just dump the table, a super-simple thing - etc. - in general, how can we better handle this communication??)

(And https://github.com/ArctosDB/internal/issues/360 would help greatly if this ends up involving big text files.)

Second option, I could try a download-only option in real-time (the error looks like your browser, not Arctos, choking - but I can also definitely build CSV that will choke the VMs, so not a completely great option).

If none of that will work for whatever reason, async is an option, albeit one that adds significant long-term expense (including aforementioned risk of choking a VM, and I haven't a clue how to evaluate any of that see eg https://github.com/ArctosDB/internal/issues/330, https://github.com/ArctosDB/internal/issues/345)

@mkoo ping re: admin-stuff