irods / irods

Open Source Data Management Software
https://irods.org
BSD 3-Clause "New" or "Revised" License
445 stars 141 forks source link

Document itree behavior around listing of very large collections #7041

Closed korydraughn closed 1 year ago

korydraughn commented 1 year ago

From https://groups.google.com/g/irod-chat/c/rOS6WqbET04/m/Mv-3TzTZBAAJ:

a user has noticed the following problem when using itree on a rather large catalog, for example:

itree --dir /<zonename>
....
regular output as expected
.....
terminating with uncaught exception of type irods::experimental::filesystem::filesystem_error: could not open collection for reading: Unknown error -310000

and on the server side:

Apr  4 16:40:13 pid:10551 NOTICE: chlGenQuery cmlGetFirstRowFromSql failure -860000
Apr  4 16:40:13 pid:10551 NOTICE: _rsGenQuery: genQuery status = -860000
Apr  4 16:40:13 pid:10551 NOTICE: rsGenQuery: _rsGenQuery failed, status = -860000
Apr  4 16:40:13 pid:10551 remote addresses: xxx, xxx ERROR: cllExecSqlWithResult: too many concurrent statements
Apr  4 16:40:13 pid:10551 NOTICE: chlGenQuery cmlGetFirstRowFromSql failure -860000
Apr  4 16:40:13 pid:10551 NOTICE: _rsGenQuery: genQuery status = -860000
Apr  4 16:40:13 pid:10551 NOTICE: rsGenQuery: _rsGenQuery failed, status = -860000
Apr  4 16:40:13 pid:10551 remote addresses: xxx, xxx ERROR: [-] /repos/irods/server/core/src/rsApiHandler.cpp:542:int readAndProcClientMsg(rsComm_t *, int) :  status [SYS_HEADER_READ_LEN_ERR]  errno [] -- message [failed to call 'read header']
        [-] /repos/irods/lib/core/src/sockComm.cpp:198:irods::error readMsgHeader(irods::network_object_ptr, msgHeader_t *, struct timeval *) :  status [SYS_HEADER_READ_LEN_ERR]  errno [] -- message [failed to call 'read header']
                [-] /repos/irods/plugins/network/tcp/libtcp.cpp:194:irods::error tcp_read_msg_header(irods::plugin_context &, void *, struct timeval *) :  status [SYS_HEADER_READ_LEN_ERR]  errno [] -- message [only read [0] of [4]]
trel commented 1 year ago

not just docs, want to handle that uncaught exception as well.

korydraughn commented 1 year ago

I believe that is already handled by @d-w-moore's latest change to itree.

If there's a permission issue, the call to determine if we're dealing with a collection will catch it.