search

data-mermaid / mermaidr

R package for accessing MERMAID authenticated API endpoints
https://data-mermaid.github.io/mermaidr/
MIT License
10 stars 1 forks source link

Error with importing data #95

Closed annacelis closed 2 months ago

annacelis commented 2 months ago

Hello! I am trying to ingest our legacy benthic PIT data. All values are matched but I keep getting this error for the dry run:

Screenshot 2024-08-23 at 5 42 18 PM

Any help is appreciated, thanks!

sharlagelfand commented 2 months ago

Hi @annacelis! Thanks for reporting this, and sorry you ran into the issue.

I believe it's related to the size of the data, and the ingestion taking too long and timing out on the server. This is something we need to fix on our end (and to have mermaidr give better messaging when this happens).

In the meantime, you can get around this by ingesting in "batches" - if you can split your data by e.g. site or date, and try ingesting each site (or each date, or however is convenient to split for you) one by one.

Please let me know if you have any questions about how to do this. And would love to know if it works for you or if you run into any additional issues!

Thanks!

annacelis commented 2 months ago

It works now! I was trying to ingest data with 12,960 observations and I'm now able to ingest them successfully in batches.

Thanks for your help!

sharlagelfand commented 2 months ago

Great, glad that worked - thank you for letting me know!