search

EDSM-NET / FrontEnd

Issues tracker for EDSM
https://www.edsm.net/
37 stars 0 forks source link

edsm.net - Journal import is very slow #518

Open Riva3000 opened 3 months ago

Riva3000 commented 3 months ago

I tried to import 100 MB of player journal files to edsm.net, using 'Import Journal' page. The import is still not finished after running for cca 16 hours.

The 1st phase of import took about 20 minutes. The second phase - 'Sending journal events...' is at 49% after the 16 hours. (I left it to run overnight to find out what is going on / get some debug info.)

I started the import by drag-droping 151 files (101 MB) at once into the form. Not sure if multiple files are supported like this, but the import started all right. Browser dev console shows no errors (aside of some advertisement .js files blocked by uBlock Origins).

Browser: Edge System: Windows 11 My EDSM.net account: https://www.edsm.net/en/user/profile/id/183063/cmdr/-Riva-

I don't know if it's not simply too much data for the system, but reporting anyway.

In case this kind of times are expected for the data volume, it would be very useful to warn user about it and eg let him cancel the import gracefully.

klightspeed commented 3 months ago

100M of journals is probably about 250k events, which at 2 events per second will take about 35 hours to upload. Note that closing the browser window will cancel the upload, and should have no adverse effects (other than needing to re-import the journals that had not yet been sent).

Riva3000 commented 3 months ago

Thanks for info @klightspeed !

As I wrote in OP, if that is the case, it would be great to explain all that to users on the import page. Because standard user of EDSM.net and the import featured will have no way of knowing and predicting that.

Another thing is why closing the window terminates the import process. I'm only guessing here but from the web page's behavior the upload of the files is first finished, and then they are processed on server side. The page would only be a progress indicator at that point. But I may very well be wrong.

klightspeed commented 3 months ago

Stage 1 is simply loading the journals into memory in Javascript, while stage 2 is actually sending the journals to EDSM.

Riva3000 commented 3 months ago

I see. Thanks.

Riva3000 commented 3 months ago

BTW is there currently a better way to upload / import the big data to EDSM ?