Uploading large files to Metalnx

[NhlamuloChauke]

Uploading 10MB or more than 10MB files and it seems to be hanging. Its says upload ... then says transferring to iRODS and it hangs there. On iRODS box i created a testfile 100MB uploads via command line no problem

[trel]

We have had a new jargon release to address larger files and parallel transfers.

We will look at an upgrade and it might fix this issue you're seeing as well.

[NhlamuloChauke]

HI Turrell,

We are currently running metalnx(version 2.5.1) which is a docker image.

Let us know when you have updated the metalnx with new jargon release

On Fri, Feb 25, 2022 at 8:16 PM Terrell Russell @.***> wrote:

We have had a new jargon release to address larger files and parallel transfers.

We will look at an upgrade and it might fix this issue you're seeing as well.

— Reply to this email directly, view it on GitHub https://github.com/irods-contrib/metalnx-web/issues/303#issuecomment-1051084332, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEGGE55ED7Z3H2BOAFFSQRLU47BQBANCNFSM5PGQBJAA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

[trel]

2.5.1 is the latest version of metalnx - so we will investigate whether your issue can be reproduced.

[trel]

Please build/test with this update to Jargon...

$ git diff
diff --git a/src/pom.xml b/src/pom.xml
index aee0008..177a67d 100644
--- a/src/pom.xml
+++ b/src/pom.xml
@@ -11,7 +11,7 @@
        <properties>

                <!-- Jargon -->
-               <jargon.version>4.3.2.2-RELEASE</jargon.version>
+               <jargon.version>4.3.2.4-RELEASE</jargon.version>

and if that doesn't fix it... we can try 4.3.2.5-SNAPSHOT

[NhlamuloChauke]

Thanks @trel I will update you

[NhlamuloChauke]

Hi @trel 4.3.2.4-RELEASE didn't resolve our issues. we had issues with 4.3.2.5-SNAPSHOT, please see the error below

[ERROR] Failed to execute goal on project metalnx-web: Could not resolve dependencies for project com.emc.metalnx:metalnx-web:war:2.5.1-RELEASE: The following artifacts could not be resolved: org.irods.jargon:jargon-extensions-if:jar:4.3.2.5-SNAPSHOT, org.irods.jargon:irodsext-data-profiler:jar:4.3.2.5-SNAPSHOT, org.irods.jargon:irodsext-data-typer:jar:4.3.2.5-SNAPSHOT, org.irods.jargon:emc-metalnx-services:jar:4.3.2.5-SNAPSHOT, org.irods.jargon:emc-metalnx-core:jar:4.3.2.5-SNAPSHOT, org.irods:irodsext-jwt-service:jar:4.3.2.5-SNAPSHOT: Could not find artifact org.irods.jargon:jargon-extensions-if:jar:4.3.2.5-SNAPSHOT in dice.repository snaps (https://raw.github.com/DICE-UNC/DICE-Maven/master/snapshots) -> [Help 1]

[trel]

Thanks for investigating. We'll work to reproduce.

[NhlamuloChauke]

Thanks, please keep us updated

[trel]

I reproduced a failure to upload with a default installation - saw this in the Metalnx logs...

metalnx_1  | 2022-03-10 02:16:05 INFO  UploadController:96 - upload()
metalnx_1  | 2022-03-10 02:16:05 INFO  UploadController:109 - multipartFile:org.springframework.web.multipart.commons.CommonsMultipartFile@294c02da
metalnx_1  | 2022-03-10 02:16:05 INFO  UploadController:67 - getUploadResponse()
metalnx_1  | 2022-03-10 02:16:05 INFO  UploadController:68 - uploadMessage:No default storage resource defined in configuration file
metalnx_1  | 2022-03-10 02:16:05 INFO  UploadController:69 - errorType:fatal
metalnx_1  | 2022-03-10 02:16:05 WARN  UploadController:83 - passing along internal server error:fatal

So I looked into the metalnx.properties file, and remembered you need to configure the default resource...

default.storage.resource=

Setting that value (first, demoResc, and then another unixfilesystem I had created) allowed me to upload successfully.

If this fixes the issue for you as well, we can make this a bug report about allowing the server to decide where files should go if the client does not have a preference.

[trel]

We discovered via browser inspection / debugging a 413 Request Entity Too Large error coming from an nginx proxy server.

This was remedied by adding to the nginx configuration:

client_max_body_size 0;

The default was 1MB and was blocking uploads until we set it to unlimited.

[NhlamuloChauke]

@trel thanks