Error occurs when registering files in nfsrods.

Narushima2030 commented 1 month ago

I get an error when registering files to nfsrods. Please tell me where the problem is.

I am running nfsrods docker on server1(10.1.19.4) and mounting nfsrods on another server:server2(10.1.19.22). On another server:server2(10.1.19.22), I mount nfsrods.

ubuntu@dms-irods-nextcloud:~$ mount |grep nfs
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
10.1.19.4:/ on /data type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=40049,timeo=600,retrans=2,sec=sys,clientaddr=10.1.19.22,local_lock=none,addr=10.1.19.4)

ubuntu@dms-irods-nextcloud:~$ ls -la testfile*
-rw-rw-r-- 1 ubuntu ubuntu 60 May 30 14:36 testfile1
-rw-rw-r-- 1 ubuntu ubuntu 57 May 30 14:36 testfile2

I try to copy the above small file to a directory on nfsrods,

Remote I/O error occurs for some files.

ubuntu@dms-irods-nextcloud:~$ sudo cp -fr ./testfile1 /data/home/rods/
ubuntu@dms-irods-nextcloud:~$ sudo cp -fr ./testfile2 /data/home/rods/
cp: cannot create regular file '/data/home/rods/testfile2': Remote I/O error

Remote I/O error occurs in testfile2, but not in testfile1. testfile2 shows Remote I/O error.

Only testfile1 is able to execute the cp command.

ubuntu@dms-irods-nextcloud:~$ sudo ls -la /data/home/rods/
total 1
drwx------ 1 root nogroup  0 May 30 14:51 .
drwx------ 1 root nogroup  0 May 30 14:38 ..
-rw------- 1 root nogroup 60 May 30 14:51 testfile1

The iRODS registration process seems to be taking place, but the file size is zero.

ubuntu@dmsirods:~$ ils -L
/rikenZone/home/rods:
  rods              0 dmsresc           60 2024-05-30.14:51 & testfile1
        generic    /home/irodsResc/home/rods/testfile1
  rods              0 dmsresc            0 2024-05-30.14:53 & testfile2
        generic    /home/irodsResc/home/rods/testfile2

The nfsrods docker is outputting the following error in the log. Do you know the cause of the problem?


2024-05-30 05:54:20.899 DEBUG Thread-42 [IRODSIdMapper] - resolveUser - _userID = 0
2024-05-30 05:54:20.899 DEBUG Thread-42 [IRODSVirtualFileSystem] - create - Creating new file [irods://rods@dmsirods.riken.jp:1247/rikenZone/home/rods/testfile3] ...
2024-05-30 05:54:20.917 ERROR Thread-42 [IRODSVirtualFileSystem] - unable to find file under path
2024-05-30 05:54:20.917 ERROR Thread-42 [NFSServerV41] - Unhandled exception:
java.io.IOException: org.irods.jargon.core.exception.FileNotFoundException: unable to find file under path
        at org.irods.nfsrods.vfs.IRODSVirtualFileSystem.create(IRODSVirtualFileSystem.java:308) ~[nfsrods.jar:?]
        at org.dcache.nfs.vfs.PseudoFs.create(PseudoFs.java:156) ~[nfsrods.jar:?]
        at org.dcache.nfs.v4.OperationOPEN.process(OperationOPEN.java:153) ~[nfsrods.jar:?]
        at org.dcache.nfs.v4.AbstractOperationExecutor.execute(AbstractOperationExecutor.java:58) ~[nfsrods.jar:?]
        at org.dcache.nfs.v4.NFSServerV41.NFSPROC4_COMPOUND_4(NFSServerV41.java:188) ~[nfsrods.jar:?]
        at org.dcache.nfs.v4.xdr.nfs4_prot_NFS4_PROGRAM_ServerStub.dispatchOncRpcCall(nfs4_prot_NFS4_PROGRAM_ServerStub.java:48) ~[nfsrods.jar:?]
        at org.dcache.oncrpc4j.rpc.RpcDispatcher$1.lambda$run$0(RpcDispatcher.java:100) ~[nfsrods.jar:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?]
        at org.dcache.oncrpc4j.rpc.RpcDispatcher$1.run(RpcDispatcher.java:99) ~[nfsrods.jar:?]
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) ~[nfsrods.jar:?]
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) ~[nfsrods.jar:?]
        at java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.irods.jargon.core.exception.FileNotFoundException: unable to find file under path
        at org.irods.jargon.core.pub.CollectionListingUtils.handleNoObjStatUnderRootOrHomeByLookingForPublicAndHome(CollectionListingUtils.java:260) ~[nfsrods.jar:?]
        at org.irods.jargon.core.pub.CollectionListingUtils.retrieveObjectStatForPath(CollectionListingUtils.java:1181) ~[nfsrods.jar:?]
        at org.irods.jargon.core.pub.CollectionAndDataObjectListAndSearchAOImpl.retrieveObjectStatForPath(CollectionAndDataObjectListAndSearchAOImpl.java:1661) ~[nfsrods.jar:?]
        at org.irods.nfsrods.vfs.IRODSVirtualFileSystem.create(IRODSVirtualFileSystem.java:298
```) ~[nfsrods.jar:?]
        ... 12 more

korydraughn commented 3 weeks ago

What version of iRODS is your NFSRODS server communicating with?

Aside from the iRODS version, I could not reproduce this against iRODS 4.3.2. Are you able to reproduce this consistently?

Are there any log messages in the iRODS server log related to the error?

Narushima2030 commented 3 weeks ago

Thank you for your reply.

The version of irods you are communicating with is 4.3.2

ubuntu@dmsirods:~$ ienv
irods_version - 4.3.2

nfsrods is built with 2.3.0.

2024-05-30 05:14:53.244 INFO Thread 1 [ServerMain] - Build time => 2024-05-22T06:09:37+0000
2024-05-30 05:14:53.244 INFO Thread 1 [ServerMain] - Build version => 2.3.0
2024-05-30 05:14:53.244 INFO Thread 1 [ServerMain] - Build SHA => ec7b053c774722ced98e033bc901c25c58167f5c
2024-05-30 05:14:53.398 INFO Thread 1 [ServerMain] - Main - Server config ==> {

Server side error log not found. I checked the irods related logs in /var/log/syslog but there don't seem to be any logs related to this error.

Sorry, I don't think I'm providing any useful information.

Is there any other information I need to provide?

korydraughn commented 3 weeks ago

Are you able to reproduce the issue consistently?

Narushima2030 commented 3 weeks ago

Yes, I will reproduce it. Check the phenomenon again.

Prepare a test file.

ubuntu@dmsirods:~$ ls -la |grep testfile
-rw-rw-r--  1 ubuntu ubuntu       34 May 30 11:44 testfile1
-rw-rw-r--  1 ubuntu ubuntu       34 May 30 14:35 testfile2
-rw-rw-r--  1 ubuntu ubuntu       34 Jun 13 08:49 testfile3

Initially, the only file registered in iRODS is testfile1.

ubuntu@dmsirods:~$ ils -L
/rikenZone/home/rods:
  rods              0 dmsresc           34 2024-06-12.09:33 & testfile1
        generic    /home/irodsResc/home/rods/testfile1

The iput command can register testfile2 without any problems.

ubuntu@dmsirods:~$ iput testfile2

ubuntu@dmsirods:~$ ils -L
/rikenZone/home/rods:
  rods              0 dmsresc           34 2024-06-12.09:33 & testfile1
        generic    /home/irodsResc/home/rods/testfile1
  rods              0 dmsresc           34 2024-06-13.08:51 & testfile2
        generic    /home/irodsResc/home/rods/testfile2

When testfile3 is placed in the nfsrods area of XXX, a Remote I/O error is output. When checking with ils, testfile3 is registered, but its size is 0.

ubuntu@dmsirods:~$ mount |grep nfs
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
10.1.19.4:/ on /data type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=40049,timeo=600,retrans=2,sec=sys,clientaddr=10.1.19.4,local_lock=none,addr=10.1.19.4)

ubuntu@dmsirods:~$ sudo cp testfile3 /data/home/rods/
cp: cannot create regular file '/data/home/rods/testfile3': Remote I/O error
ubuntu@dmsirods:~$ ils -L
/rikenZone/home/rods:
  rods              0 dmsresc           34 2024-06-12.09:33 & testfile1
        generic    /home/irodsResc/home/rods/testfile1
  rods              0 dmsresc           34 2024-06-13.08:51 & testfile2
        generic    /home/irodsResc/home/rods/testfile2
  rods              0 dmsresc            0 2024-06-13.08:52 & testfile3
        generic    /home/irodsResc/home/rods/testfile3

korydraughn commented 2 weeks ago

I still haven't been able to reproduce the issue.

I have some questions for you.

Is the issue reproducible without using sudo?
Is the issue specific to the rods user account? (sudo is viewed as the rods account)
Is there anything special about the environment (i.e. client, NFSRODS, and iRODS server)?

Please show the resource information for dmsresc using ilsresc dmsresc.

korydraughn commented 2 weeks ago

Also, please show the permissions on the home collection for rods (i.e. ils -A /rikenZone/home/rods).

Narushima2030 commented 2 weeks ago

Thank you for your reply.

Is the issue reproducible without using sudo?

An error is output with or without sudo.

ubuntu@dmsirods:~$ cp testfile3 /data/home/rods/
cp: cannot stat '/data/home/rods/testfile3': Remote I/O error

ubuntu@dmsirods:~$ sudo cp testfile3 /data/home/rods/
cp: cannot stat '/data/home/rods/testfile3': Remote I/O error

Is the issue specific to the rods user account? (sudo is viewed as the rods account)

I copied a file to the nfsrods directory (/data/home/www-data/) using a different account, but I got the same error message.

www-data@dmsirods:/local/home/ubuntu$ ls -la testfile*
-rw-rw-r-- 1 ubuntu   ubuntu   34 May 30 11:44 testfile1
-rwxrwxrwx 1 ubuntu   ubuntu   34 May 30 14:35 testfile2
-rw-rw-r-- 1 ubuntu   ubuntu   34 Jun 13 08:49 testfile3
-rwxrwxrwx 1 www-data www-data 34 Jun 20 11:05 testfile4

www-data@dmsirods:/local/home/ubuntu$ cp ./testfile1 /data/home/www-data/
cp: cannot create regular file '/data/home/www-data/testfile1': Remote I/O error
www-data@dmsirods:/local/home/ubuntu$ cp ./testfile2 /data/home/www-data/
cp: cannot create regular file '/data/home/www-data/testfile2': Remote I/O error
www-data@dmsirods:/local/home/ubuntu$ cp ./testfile3 /data/home/www-data/
cp: cannot create regular file '/data/home/www-data/testfile3': Remote I/O error
www-data@dmsirods:/local/home/ubuntu$ cp ./testfile4 /data/home/www-data/
cp: cannot create regular file '/data/home/www-data/testfile4': Remote I/O error

ubuntu@dmsirods:~$ ils -L
/rikenZone/home/www-data:
  www-data          0 dmsresc            0 2024-06-20.16:41 & testfile1
        generic    /home/irodsResc/home/www-data/testfile1
  www-data          0 dmsresc            0 2024-06-20.16:43 & testfile2
        generic    /home/irodsResc/home/www-data/testfile2
  www-data          0 dmsresc            0 2024-06-20.16:43 & testfile3
        generic    /home/irodsResc/home/www-data/testfile3
  www-data          0 dmsresc            0 2024-06-20.16:43 & testfile4
        generic    /home/irodsResc/home/www-data/testfile4

Is there anything special about the environment (i.e. client, NFSRODS, and iRODS server)?

The differences from when using nfsrods without any problems are:

irods and nfsrods versions
irods DB changed from postgresSQL to MariaDB (To support High Availability)

Also, please show the permissions on the home collection for rods (i.e. ils -A /rikenZone/home/rods).

The information for dmsresc is as follows:

ubuntu@dmsirods:~$ ils -A /rikenZone/home/rods
/rikenZone/home/rods:
        ACL - rods#rikenZone:own   
        Inheritance - Disabled
  testfile1
        ACL - rods#rikenZone:own   
  testfile2
        ACL - rods#rikenZone:own   
  testfile3
        ACL - rods#rikenZone:own

ubuntu@dmsirods:~$ ilsresc -l dmsresc
resource name: dmsresc
id: 10013
zone: rikenZone
type: unixfilesystem
location: dmsirods.riken.jp
vault: /home/irodsResc
free space: 
free space time: : Never
status: 
info: 
comment: 
create time: 01714111611: 2024-04-26.15:06:51
modify time: 01714111611: 2024-04-26.15:06:51
context: 
parent: 
parent context:

Could there be a problem with my account or file permissions? I would be grateful if you could give me some examples of correct account information and file permissions.

irods / irods_client_nfsrods

Error occurs when registering files in nfsrods. #207