search

pepkit / pephub

A web API and database for biological sample metadata
https://pephub.databio.org
BSD 2-Clause "Simplified" License
12 stars 2 forks source link

PEPhub fails for large PEPs #225

Closed nleroy917 closed 3 days ago

nleroy917 commented 10 months ago

I was trying to upload a very large sample table. It had >75,000 samples, with 1875(I think?) attributes per sample. The file was ~325 megabytes.

PEPhub responded with this toast message:

pephub error message

Two things I bring up:

  1. Should PEPhub support very large files like this?
  2. If not, can we catch the 413 and render a more user-friendly toast message?
khoroshevskyi commented 10 months ago

75K is very huge PEP. I think PEPhub server can fail if you will try to open this pep using peppy. I would say, let's not do this. At this point max pep should be less then 10K

nleroy917 commented 10 months ago

Yeah I agree. I say we just catch the 413 and notify the user there's a max file size limit.

nsheff commented 10 months ago

I agree that 75k is probably outside the scope of what we were thinking.

I'm assuming this was embeddings? Maybe we need a separate service with database of embeddings?

nleroy917 commented 10 months ago

It was actually GEO_GSM_complete

khoroshevskyi commented 10 months ago

It was actually GEO_GSM_complete

This file is more than 300 MB, it's too huge + too many columns.

khoroshevskyi commented 10 months ago

In the future, I think it will be possible to store such a big files if we will develop new way for getting samples, but for right now it is impossible

nsheff commented 1 month ago

I'm working with a group that wants to hold a large table, maybe on the scale of 10,000 records.

He pasted 2000 records into the table, and the javascript could handle it, but when he clicked 'Save' it said "Unknown Error Occurred".

with 1000 rows, it worked.

Can we clarify this?

khoroshevskyi commented 3 days ago

This issue was solved in pephub 0.11.9