search

PASTAplus / DataPortal

EDI Data Portal
3 stars 3 forks source link

Desktop upload of 5.8GB package results in "No value specified for request parameter 'metadataSource'" exception #139

Closed servilla closed 2 months ago

servilla commented 2 months ago

After allowing desktop uploads for unregulated volumes, portal-s.edirepository.org returned a java.lang.IllegalStateException: No value was specified for the request parameter 'metadataSource' exception. The total data volume was 5.8GB, and one CSV file was 5.6GB.

servilla commented 2 months ago

File manifest:

total 5,948,866,560
edi.1118.1.xml                4,519,994
fish.csv                  5,753,522,296
fishsurvey_compressed.rda    31,294,364
Length_conversions.csv              745 
survey.csv                  159,509,093
servilla commented 2 months ago

Problem view:

diagram-export-7-3-2024-8_50_23-PM

Fixes:

diagram-export-7-3-2024-8_50_55-PM

/etc/nginx/sites-enabled/portal*.nginx
proxy_request_buffering off;
/etc/tomcat9/server.xml
<Connector port="8080" protocol="HTTP/1.1"
           maxPostSize="-1"
           maxSavePostSize="-1"
           connectionTimeout="60000"
           disableUploadTimeout="false"
           keepAliveTimeout="3000000"
           redirectPort="8443" />
/home/pasta/git/DataPortal/Webroot/WEB-INF/web.xml
<!-- Multipart-config -->
<multipart-config>
   <max-file-size>10737418240</max-file-size>
   <max-request-size>10737418240</max-request-size>
   <file-size-threshold>0</file-size-threshold>
</multipart-config>
servilla commented 2 months ago

Tested the above data package on portal-s (2024-07-4), which succeeded past the point where the error occurred. Job did not complete due to permissions on package identifier reservation.

Note that operational storage requires 2x the size of the data package.