search

ePADD / epadd

ePADD is a software package developed by Stanford University's Special Collections & University Archives that supports archival processes around the appraisal, ingest, processing, discovery, and delivery of email archives.
https://www.epaddproject.org
111 stars 24 forks source link

Took too long for some processes after using Bagit spec #233

Closed peterchanws closed 6 years ago

peterchanws commented 6 years ago

version epadd-standalone-beta-25June.jar 32GB RAM i7-5820K CPU 6 Cores; Win 10; Took 20 min to close Bush full in processing module 16GB RAM i7-8550U CPU 4 Cores; Win 10; Took 22 min to close Bush full in processing module

version epadd-standalone-v51.jar 12GB RAM Xeon CPU 4 Cores; MacOS 10.12 Took 20 sec to close Bush full in processing module

peterchanws commented 6 years ago

ver 6-beta1-3July Took 27 min to close Bush full in processing module

peterchanws commented 6 years ago

ver v6-beta1-3July Took 10 min to add full Bush to processing

peterchanws commented 6 years ago

The time taken to add a collection in processing is determined by the size of that archive. During this process the archive folder is copied to epadd-processing. For Bush full the size of the folder is around 6 GB so the copy of this much data takes around 10 min.

Earlier when closing the archive the index was also getting saved. We are now saving only mutable metadata of an archive. Moreover, we display progress bar to the user now.

peterchanws commented 6 years ago

Ver 6 beta 2, Bush full

Win 10, 16GB RAM Took 18 min to verify checksum under - acceptable Took 5:33 min to close archive in processing

Win 10, 32 GB RAM Took 1:52 min to close archive in processing

OS 10.13, 16GB RAM Took 2 min to import to processing Took 11 min to verify checksum under - acceptable Took 30 sec to close archive in processing

OS 13.13 8GB RAM Took 30 sec to close archive in processing

Close under Win take long time (5 or 2 min); under Mac <1 min Anything we can do for Windows?

Remind users to clearing cache when using the new vresion

peterchanws commented 6 years ago

July 17, 2018 version Took 8 seconds under Win, 6 seconds under OSX. Great achievement!!

peterchanws commented 6 years ago

@chinuhub July 20, 2018 version After I click Close for Terry full in porcessing module (win 10 64GB ram), system shown "Saving Archive" for 16 min and then show Sorry, there was an error, no JSON was returned! Please retry and if the error persists, report the problem to us.

peterchanws commented 6 years ago

This issue is described in technical terms by Chinmay in #239 Close this issue

peterchanws commented 6 years ago

refer to #239