search

irods-contrib / metalnx-web

Metalnx Web Application
https://metalnx.github.io/
BSD 3-Clause "New" or "Revised" License
36 stars 36 forks source link

Unable to Upload Files to IRODS (Metalnx?) #354

Closed tenajsystems closed 7 months ago

tenajsystems commented 8 months ago

I am unable to upload files to irods. Several attempts I get the attached error.

1- transfer to iRods 2- error message

We are running iRODS Version 4.3.1 and Metalnx, 2.5.1. Restarting the irods service was "successful" but checking the status of the service shows some validation issues as shown below (not sure if that is the cause). I have checked the syntax server_config.json, version.json and irods_environment.json with a json parser and all came back as valid jason output. I am new to irods and matalnx so any assistance troubleshooting will be greatly appreciated.

[root@irods-server ~]# systemctl status irods.service
● irods.service - LSB: Manages the iRODS service
   Loaded: loaded (/etc/rc.d/init.d/irods; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2023-11-08 08:18:22 EST; 6s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6574 ExecStop=/etc/rc.d/init.d/irods stop (code=exited, status=0/SUCCESS)
  Process: 6604 ExecStart=/etc/rc.d/init.d/irods start (code=exited, status=0/SUCCESS)

Nov 08 08:18:21 irods-server  irods[6604]: Traceback (most recent call last):
Nov 08 08:18:21 irods-server  irods[6604]: File "/var/lib/irods/scripts/irods/controller.py", line 69, in start
Nov 08 08:18:21 irods-server  irods[6604]: self.config.validate_configuration()
Nov 08 08:18:21 irods-server  irods[6604]: File "/var/lib/irods/scripts/irods/configuration.py", line 296, in validate_configuration
Nov 08 08:18:21 irods-server  irods[6604]: lib.indent(*skipped)))
Nov 08 08:18:21 irods-server  irods[6604]: irods.exceptions.IrodsWarning: Skipped validation for the following files:
Nov 08 08:18:21 irods-server  irods[6604]: /etc/irods/server_config.json
Nov 08 08:18:21 irods-server  irods[6604]: /var/lib/irods/version.json
Nov 08 08:18:21 irods-server  irods[6604]: /var/lib/irods/.irods/irods_environment.json
Nov 08 08:18:22 irods-server  systemd[1]: Started LSB: Manages the iRODS service.
korydraughn commented 8 months ago

For the upload failure, try looking at the log file of the iRODS server and the Metalnx log output to see if there's anything in there related to the failed upload.

For the schema validation failure, check /var/lib/irods/log/control_log.txt. That should contain the reason behind the failure.

You mentioned you're running Metalnx 2.5.1. Consider upgrading to 2.6.1 (released in March 2023). Before upgrading, please make sure you test it using a test environment.

tenajsystems commented 8 months ago

For the upload failure, try looking at the log file of the iRODS server and the Metalnx log output to see if there's anything in there related to the failed upload.

For the schema validation failure, check /var/lib/irods/log/control_log.txt. That should contain the reason behind the failure.

You mentioned you're running Metalnx 2.5.1. Consider upgrading to 2.6.1 (released in March 2023). Before upgrading, please make sure you test it using a test environment.

@korydraughn, thank you! we were able to get the validation issues fixed following the guidance provided here: https://github.com/irods/irods/issues/7414 . The file upload still fails. What is the name of the iRODS server log and Metalnx log that I need to be looking at? Thanks!

korydraughn commented 8 months ago

The iRODS server log is /var/log/irods/irods.log.

If you're running Metalnx in a docker container, you can view its logs by running the following:

docker logs -f <metalnx_container_name>
chip-aws commented 7 months ago

We've updated to the latest version of Metalnx (Metalnx Version | 2.6.1-RELEASE) and we're running iRODS 4.3.1-0, but we're still unable to upload files using Metalnx. Below are the results of following the Metalnx Docker container logs while attempting a file upload of a file named testfile.txt which was only 369 bytes in size. It appears to keep stating that the file doesn't exist:

selectFieldSource:DEFINED_QUERY_FIELD], conditions=[GenQueryBuilderCondition selectFieldSource:DEFINED_QUERY_FIELD selectFieldNumericTranslation:301 selectFieldColumnName:RESC_ID operator:EQUAL value:'1087779'], orderByFields=[], distinct=true, upperCase=false, computeTotalRowCount=false]] 2023-11-16 21:02:04 INFO GenQueryProcessor:85 - continuation value: 0 2023-11-16 21:02:04 INFO GenQueryProcessor:95 - total records:0 2023-11-16 21:02:04 INFO QueryResultProcessingUtils:57 - rows returned from iRODS query: 1 2023-11-16 21:02:04 INFO GenQueryProcessor:116 - auto closing result set 2023-11-16 21:02:04 INFO GenQueryProcessor:145 - closeResults() 2023-11-16 21:02:04 INFO GenQueryProcessor:151 - no results to close, ignore 2023-11-16 21:02:04 WARN ResourceAOImpl:621 - unable to format resourceFreeSpace for value: setting to 0 2023-11-16 21:02:04 INFO ResourceServiceImpl:118 - getDataGridResource() 2023-11-16 21:02:04 INFO ResourceServiceImpl:118 - getDataGridResource() 2023-11-16 21:02:04 INFO ResourceServiceImpl:118 - getDataGridResource() 2023-11-16 21:02:04 INFO ResourceServiceImpl:118 - getDataGridResource() 2023-11-16 21:02:04 INFO ResourceServiceImpl:118 - getDataGridResource() 2023-11-16 21:02:04 INFO HttpResponseHandlerInterceptor:52 - postHandle() 2023-11-16 21:02:04 INFO ConfigServiceImpl:128 - getGlobalConfig() 2023-11-16 21:02:04 INFO LoggedUserUtils:29 - getLoggedDataGridUser() 2023-11-16 21:02:04 INFO LoggedUserUtils:38 - auth:org.springframework.security.authentication.UsernamePasswordAuthenticationToken@848764d8: Principal: dfwhite; Credentials: [PROTECTED]; Authenticated: true; Details: com.emc.metalnx.services.auth.UserTokenDetails@7f4f86d7; Granted Authorities: com.emc.metalnx.services.auth.IRODSAuthenticationProvider$IRODSAdminGrantedAuthority@5a8c5d2c 2023-11-16 21:03:25 INFO UploadController:96 - upload() 2023-11-16 21:03:25 INFO UploadController:109 - multipartFile:org.springframework.web.multipart.commons.CommonsMultipartFile@545a87b3 2023-11-16 21:03:25 INFO UploadController:126 - parsed parameters... 2023-11-16 21:03:25 INFO UploadServiceImpl:65 - upload() 2023-11-16 21:03:25 INFO UploadServiceImpl:73 - file:org.springframework.web.multipart.commons.CommonsMultipartFile@545a87b3 2023-11-16 21:03:25 INFO UploadServiceImpl:74 - targetPath:/lifelibZone/home/dfwhite/uploads 2023-11-16 21:03:25 INFO UploadServiceImpl:75 - computeCheckSum:false 2023-11-16 21:03:25 INFO UploadServiceImpl:76 - replicateFile:false 2023-11-16 21:03:25 INFO UploadServiceImpl:77 - replicationResc:null 2023-11-16 21:03:25 INFO UploadServiceImpl:78 - destResc:lifetimelib-rescResource 2023-11-16 21:03:25 INFO UploadServiceImpl:79 - overwrite:false 2023-11-16 21:03:25 INFO UploadServiceImpl:90 - Setting default resource to lifetimelib-rescResource 2023-11-16 21:03:25 INFO IRODSFileFactoryImpl:119 - instanceIRODSFile() 2023-11-16 21:03:25 INFO IRODSFileFactoryImpl:129 - parent:/lifelibZone/home/dfwhite/uploads 2023-11-16 21:03:25 INFO IRODSFileFactoryImpl:130 - child:testfile.txt 2023-11-16 21:03:25 INFO IRODSFileImpl:273 - setting file name, given path = testfile.txt 2023-11-16 21:03:25 INFO IRODSFileImpl:274 - detected local separator = / 2023-11-16 21:03:25 INFO IRODSFileImpl:303 - file name was set as: testfile.txt 2023-11-16 21:03:25 INFO UploadServiceImpl:110 - targetFile:irods://dfwhite@[redacted]:2247/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:340 - isFileExists() 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:346 - checking existence of: /lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:432 - getObjStat(final String irodsAbsolutePath) 2023-11-16 21:03:25 INFO CollectionAndDataObjectListAndSearchAOImpl:1628 - retrieveObjectStatForPathWithHeuristicPathGuessing() 2023-11-16 21:03:25 INFO CollectionListingUtils:1180 - got a file not found, try to heuristically produce an objstat 2023-11-16 21:03:25 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-16 21:03:25 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO CollectionAndDataObjectListAndSearchAOImpl:1640 - got a file not found, try to heuristically produce an objstat 2023-11-16 21:03:25 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-16 21:03:25 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:356 - file not found, will treat as not exists 2023-11-16 21:03:25 INFO Stream2StreamAOImpl:122 - transferStreamToFile(), inputStream:java.io.ByteArrayInputStream@2ad54781 2023-11-16 21:03:25 INFO Stream2StreamAOImpl:123 - targetFile:irods://dfwhite@[redacted]:2247/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO Stream2StreamAOImpl:131 - adding buffer around input stream 2023-11-16 21:03:25 INFO Stream2StreamAOImpl:144 - target file is an iRODS file 2023-11-16 21:03:25 INFO Stream2StreamAOImpl:151 - not using transfer redirects, so do not do any stream re-routing 2023-11-16 21:03:25 INFO IRODSFileFactoryImpl:199 - instanceIRODSFileOutputStream() 2023-11-16 21:03:25 INFO IRODSFileFactoryImpl:217 - instanceIRODSFileOutputStream() 2023-11-16 21:03:25 INFO IRODSFileFactoryImpl:227 - file:irods://dfwhite@[redacted]:2247/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileFactoryImpl:228 - openFlags:WRITE 2023-11-16 21:03:25 INFO IRODSFileOutputStream:138 - openIRODSFile() 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:340 - isFileExists() 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:346 - checking existence of: /lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:432 - getObjStat(final String irodsAbsolutePath) 2023-11-16 21:03:25 INFO CollectionAndDataObjectListAndSearchAOImpl:1628 - retrieveObjectStatForPathWithHeuristicPathGuessing() 2023-11-16 21:03:25 INFO CollectionListingUtils:1180 - got a file not found, try to heuristically produce an objstat 2023-11-16 21:03:25 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-16 21:03:25 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO CollectionAndDataObjectListAndSearchAOImpl:1640 - got a file not found, try to heuristically produce an objstat 2023-11-16 21:03:25 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-16 21:03:25 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:356 - file not found, will treat as not exists 2023-11-16 21:03:25 INFO IRODSFileOutputStream:144 - exists? false 2023-11-16 21:03:25 INFO IRODSFileOutputStream:164 - file does not exist, create it by adjusting open flags 2023-11-16 21:03:25 INFO IRODSFileImpl:1306 - open() 2023-11-16 21:03:25 INFO IRODSFileImpl:1318 - openFlags:READ_WRITE_CREATE_IF_NOT_EXISTS 2023-11-16 21:03:25 INFO IRODSFileImpl:1319 - coordinated:false 2023-11-16 21:03:25 INFO IRODSFileImpl:1324 - openWithMode() 2023-11-16 21:03:25 INFO IRODSFileImpl:1326 - openFlags:READ_WRITE_CREATE_IF_NOT_EXISTS 2023-11-16 21:03:25 INFO IRODSFileImpl:1327 - coordinated:false 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:794 - openFile(final IRODSFile irodsFile,final DataObjInp.OpenFlags openFlags) 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:800 - irodsFile:irods://dfwhite@[redacted]:2247/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:801 - openFlags:READ_WRITE_CREATE_IF_NOT_EXISTS 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:432 - getObjStat(final String irodsAbsolutePath) 2023-11-16 21:03:25 INFO CollectionAndDataObjectListAndSearchAOImpl:1628 - retrieveObjectStatForPathWithHeuristicPathGuessing() 2023-11-16 21:03:25 INFO CollectionListingUtils:1180 - got a file not found, try to heuristically produce an objstat 2023-11-16 21:03:25 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-16 21:03:25 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO CollectionAndDataObjectListAndSearchAOImpl:1640 - got a file not found, try to heuristically produce an objstat 2023-11-16 21:03:25 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-16 21:03:25 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:340 - isFileExists() 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:346 - checking existence of: /lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:432 - getObjStat(final String irodsAbsolutePath) 2023-11-16 21:03:25 INFO CollectionAndDataObjectListAndSearchAOImpl:1628 - retrieveObjectStatForPathWithHeuristicPathGuessing() 2023-11-16 21:03:25 INFO CollectionListingUtils:1180 - got a file not found, try to heuristically produce an objstat 2023-11-16 21:03:25 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-16 21:03:25 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO CollectionAndDataObjectListAndSearchAOImpl:1640 - got a file not found, try to heuristically produce an objstat 2023-11-16 21:03:25 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-16 21:03:25 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/uploads/testfile.txt 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:356 - file not found, will treat as not exists 2023-11-16 21:03:25 INFO IRODSFileSystemAOImpl:922 - opening file:/lifelibZone/home/dfwhite/uploads/testfile.txt

korydraughn commented 7 months ago

I've reproduced this issue.

We'll get this resolved asap.

korydraughn commented 7 months ago

I just tried this again and was able to upload a file after setting default.storage.resource to a valid resource.

Please try to upload another file and check the logs of the iRODS server which Metalnx is connected to. The iRODS server log can be found at /var/log/irods/irods.log if you've configured syslog to write it there. If you haven't configured syslog to write iRODS specific messages to a separate log file, check the syslog file of your OS for iRODS messages.

chip-aws commented 7 months ago

Hi Kory,

We still appear to be getting the same error. During the test, I updated default.storage.resource in metalnx.properties to LTLSils which is one of our valid resources, rebooted the server, and then logged in as dfwhite which is a rodsadmin, but the same error still occurred:

iadmin>lr LTLSils bundleResc LTLRepl LTLRenci LTLResc lifetimelib-rescResource

iadmin>lu dfwhite user_id: [redacted] user_name: dfwhite user_type_name: rodsadmin zone_name: lifelibZone user_info:

Results from docker logs -f metalnx while attempting a file upload:

023-11-28 18:36:39 INFO UploadController:96 - upload() 2023-11-28 18:36:39 INFO UploadController:109 - multipartFile:org.springframework.web.multipart.commons.CommonsMultipartFile@558ada3b 2023-11-28 18:36:39 INFO UploadController:126 - parsed parameters... 2023-11-28 18:36:39 INFO UploadServiceImpl:65 - upload() 2023-11-28 18:36:39 INFO UploadServiceImpl:73 - file:org.springframework.web.multipart.commons.CommonsMultipartFile@558ada3b 2023-11-28 18:36:39 INFO UploadServiceImpl:74 - targetPath:/lifelibZone/home/dfwhite 2023-11-28 18:36:39 INFO UploadServiceImpl:75 - computeCheckSum:false 2023-11-28 18:36:39 INFO UploadServiceImpl:76 - replicateFile:false 2023-11-28 18:36:39 INFO UploadServiceImpl:77 - replicationResc:undefined 2023-11-28 18:36:39 INFO UploadServiceImpl:78 - destResc:LTLSils 2023-11-28 18:36:39 INFO UploadServiceImpl:79 - overwrite:false 2023-11-28 18:36:39 INFO UploadServiceImpl:90 - Setting default resource to LTLSils 2023-11-28 18:36:39 INFO IRODSFileFactoryImpl:119 - instanceIRODSFile() 2023-11-28 18:36:39 INFO IRODSFileFactoryImpl:129 - parent:/lifelibZone/home/dfwhite 2023-11-28 18:36:39 INFO IRODSFileFactoryImpl:130 - child:test.txt 2023-11-28 18:36:39 INFO IRODSFileImpl:273 - setting file name, given path = test.txt 2023-11-28 18:36:39 INFO IRODSFileImpl:274 - detected local separator = / 2023-11-28 18:36:39 INFO IRODSFileImpl:303 - file name was set as: test.txt 2023-11-28 18:36:39 INFO UploadServiceImpl:110 - targetFile:irods://dfwhite@redacted:2247/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:340 - isFileExists() 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:346 - checking existence of: /lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:432 - getObjStat(final String irodsAbsolutePath) 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1628 - retrieveObjectStatForPathWithHeuristicPathGuessing() 2023-11-28 18:36:39 INFO CollectionListingUtils:1180 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1640 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:356 - file not found, will treat as not exists 2023-11-28 18:36:39 INFO Stream2StreamAOImpl:122 - transferStreamToFile(), inputStream:java.io.ByteArrayInputStream@401836f7 2023-11-28 18:36:39 INFO Stream2StreamAOImpl:123 - targetFile:irods://dfwhite@redacted:2247/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO Stream2StreamAOImpl:131 - adding buffer around input stream 2023-11-28 18:36:39 INFO Stream2StreamAOImpl:144 - target file is an iRODS file 2023-11-28 18:36:39 INFO Stream2StreamAOImpl:151 - not using transfer redirects, so do not do any stream re-routing 2023-11-28 18:36:39 INFO IRODSFileFactoryImpl:199 - instanceIRODSFileOutputStream() 2023-11-28 18:36:39 INFO IRODSFileFactoryImpl:217 - instanceIRODSFileOutputStream() 2023-11-28 18:36:39 INFO IRODSFileFactoryImpl:227 - file:irods://dfwhite@redacted.ils.unc.edu:2247/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileFactoryImpl:228 - openFlags:WRITE 2023-11-28 18:36:39 INFO IRODSFileOutputStream:138 - openIRODSFile() 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:340 - isFileExists() 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:346 - checking existence of: /lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:432 - getObjStat(final String irodsAbsolutePath) 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1628 - retrieveObjectStatForPathWithHeuristicPathGuessing() 2023-11-28 18:36:39 INFO CollectionListingUtils:1180 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1640 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:356 - file not found, will treat as not exists 2023-11-28 18:36:39 INFO IRODSFileOutputStream:144 - exists? false 2023-11-28 18:36:39 INFO IRODSFileOutputStream:164 - file does not exist, create it by adjusting open flags 2023-11-28 18:36:39 INFO IRODSFileImpl:1306 - open() 2023-11-28 18:36:39 INFO IRODSFileImpl:1318 - openFlags:READ_WRITE_CREATE_IF_NOT_EXISTS 2023-11-28 18:36:39 INFO IRODSFileImpl:1319 - coordinated:false 2023-11-28 18:36:39 INFO IRODSFileImpl:1324 - openWithMode() 2023-11-28 18:36:39 INFO IRODSFileImpl:1326 - openFlags:READ_WRITE_CREATE_IF_NOT_EXISTS 2023-11-28 18:36:39 INFO IRODSFileImpl:1327 - coordinated:false 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:794 - openFile(final IRODSFile irodsFile,final DataObjInp.OpenFlags openFlags) 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:800 - irodsFile:irods://dfwhite@redacted:2247/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:801 - openFlags:READ_WRITE_CREATE_IF_NOT_EXISTS 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:432 - getObjStat(final String irodsAbsolutePath) 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1628 - retrieveObjectStatForPathWithHeuristicPathGuessing() 2023-11-28 18:36:39 INFO CollectionListingUtils:1180 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1640 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:340 - isFileExists() 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:346 - checking existence of: /lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:432 - getObjStat(final String irodsAbsolutePath) 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1628 - retrieveObjectStatForPathWithHeuristicPathGuessing() 2023-11-28 18:36:39 INFO CollectionListingUtils:1180 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1640 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:356 - file not found, will treat as not exists 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:922 - opening file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileImpl:273 - setting file name, given path = /lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileImpl:274 - detected local separator = / 2023-11-28 18:36:39 INFO IRODSFileImpl:303 - file name was set as: test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:1093 - fileDeleteForce(final IRODSFile irodsFile) 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:1098 - deleting:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileImpl:711 - isFile() for path:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileSystemAOImpl:432 - getObjStat(final String irodsAbsolutePath) 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1628 - retrieveObjectStatForPathWithHeuristicPathGuessing() 2023-11-28 18:36:39 INFO CollectionListingUtils:1180 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO CollectionAndDataObjectListAndSearchAOImpl:1640 - got a file not found, try to heuristically produce an objstat 2023-11-28 18:36:39 INFO CollectionListingUtils:172 - handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome() 2023-11-28 18:36:39 INFO CollectionListingUtils:259 - really is a not found for file:/lifelibZone/home/dfwhite/test.txt 2023-11-28 18:36:39 INFO IRODSFileImpl:727 - file not found 2023-11-28 18:36:39 ERROR IRODSFileSystemAOImpl:1102 - file delete, given irodsFile is not a file 2023-11-28 18:36:39 ERROR FileOperationServiceImpl:190 - Could not delete data object: file delete, given irodsFile is not a file 2023-11-28 18:36:39 ERROR UploadServiceImpl:142 - Upload stream failed from Metalnx to the data grid. DIRECT_CHILD_ACCESS 2023-11-28 18:36:39 ERROR UploadController:145 - DataGridException uploading file

There was no additional output from docker logs -f metalnx-database, or in the content of the control_log.txt file on the system located in /var/lib/irods/log/.

For tail -f /var/log/messages, this line appeared:

Nov 28 13:41:10 redacted irodsServer[3446]: {"log_category":"legacy","log_level":"error","log_message":"[rsDataObjOpen_impl:920] - [DIRECT_CHILD_ACCESS: key_word contains child resource\n\n] [error_code=[-1816000], path=[/lifelibZone/home/dfwhite/test.txt], hierarchy=[]","request_api_name":"DATA_OBJ_OPEN_AN","

trel commented 7 months ago

DIRECT_CHILD_ACCESS is the clue here...

What is the output of $ ilsresc?

You need to point metalnx at the root of the hierarchy that you want the uploaded files to go to - not a child resource farther down in the hierarchy.

chip-aws commented 7 months ago

It looks like this: 

lifetimelib-rescResource:unixfilesystem
LTLResc:passthru
└── LTLRepl:replication
    ├── LTLRenci:unixfilesystem
    └── LTLSils:unixfilesystem
chip-aws commented 7 months ago

I did try pointing it to LTLResc earlier, but I still had the same error.

chip-aws commented 7 months ago

At least the upload still failed.

trel commented 7 months ago

I didn't think that LTLRenci was still online? If you pull that out of the tree, I think it might behave better...

Try... iadmin rmchildfromresc LTLRepl LTLRenci

Then point metalnx to LTLResc...

If needed, you can add it back easily with iadmin addchildtoresc LTLRepl LTLRenci

chip-aws commented 7 months ago

That makes sense. I'll make that change and point Metalnx to the root.

chip-aws commented 7 months ago

Cool! That fixed that issue. I can upload files in Metalnx now.

[dfwhite@server ~]$ ilsresc lifetimelib-rescResource:unixfilesystem LTLRenci:unixfilesystem LTLResc:passthru └── LTLRepl:replication └── LTLSils:unixfilesystem

chip-aws commented 7 months ago

I do have another issue I noticed a while back that still seems to be an issue - I can't delete files. Should I post that in a new issue? Thanks for the help with the uploads though. (Ignore this as a few browser refreshes seems to have fixed this issue too).

trel commented 7 months ago

Glad you got the upload working. Closing.