question about test script.

[yr-wang-hit]

I have tried irods_capability_automated_ingest tools by hand with the help of tutorials and it ran smoothly. BUT it cannot pass through the test script, aka python -m irods_capability_automated_ingest.test.test_irods_sync the error report said something like FileNotFoundError: [Errno 2] No such file or directory: 'irods_capability_automated_ingest.examples.append_json' it can work if I specify custom event_handlers with a py file path as follows. python -m irods_capability_automated_ingest.irods_sync start ~/lz_test/src_dir /zone1/home/rods/reg_coll --event_handler ./examples/append_json.py --synchronous --progress But It cannot recogonize the file path if we change the path to something like irods_capability_automated_ingest.examples.append_json Unfortunately, all the test cases use the second way.

I have updated the code so I install the python module with "python setup.py install", I cannot figure the problems here and ask for help. Thanks a lot.

[alanking]

The specific problem that you're encountering is likely due to a recent-ish change in which the event handler is read in from a file which is copied into Redis for use by the Celery workers: https://github.com/irods/irods_capability_automated_ingest/commit/faffe29be09f8ee60e76a9cb6b8979925225c1a0

This change implicitly changed the --event-handler option such that it takes a path to a python file rather than a dot-delimited python module. The tests were updated in this commit to account for the change: https://github.com/irods/irods_capability_automated_ingest/commit/d4c9be38b3d547746a772374a6db3c54c09a8b4d

Please confirm that you are running at least commit d4c9be38b3d547746a772374a6db3c54c09a8b4d

It's possible that we need to update the documentation, as well. @d-w-moore and I have also discussed making the option more flexible such that a module can be passed and still provide the intended behavior. In any case, using the path to the file instead of the dot-delimited python module should work.

In closing, I'll say that running the tests requires a very specific setup that I personally find is very easy to get wrong. I would recommend using the docker-compose file to run the tests as described here: https://github.com/irods/irods_capability_automated_ingest/tree/main/docker/ingest-test#how-to-run-the-test-suite-using-docker-compose To change what version and tests are run, modify the appropriate environment variables in icommands.env, which is passed in the docker-compose up invocation.

[yr-wang-hit]

Thanks alan and sorry for late response, I will try the docker-compose advice, But I feel very confused that when running the same test script twice, the result is different(One ALL pass and the other nearly ALL error) ,and one test case failure can pollute the test environment and other tests will report error, too. A very tricky and severe situation during running test script is irods.exception.CAT_COLLECTION_NOT_EMPTY: None It looks like the test script can sometimes create incomplete files and irods cannot delete them. remote addresses: x.x.x.x ERROR: rmUtil: rm error for /zone1/home/rods/a_remote, status = -821000 status = -821000 CAT_COLLECTION_NOT_EMPTY in the situation, the ils can list the files but the ils -L cannot see them. What is happenning?

Thanks a lot.

[alanking]

I have found that the directory used for testing the filesystem scanner sometimes does not get cleaned up appropriately. If you remove the directory and try again, I would expect it to work. Checking for and removing any existing directory which is to act as the "filesystem" to scan should probably be part of the setup of the test.

[d-w-moore]

In reference to @alanking 's earlier post, yes we could / should probably restore the dotted module-name syntax, but allow either (switching on whether a '/' is in the name or not). I've been experimenting with putting compiled objects (ie bytecode) in the key-value store, which could fairly easily be reconstituted 'on the other side' and it seems feasible.

[alanking]

I think we can close this. Please open a new issue if there are any further questions about running tests.

Some concluding thoughts, for posterity...

The tests should be run in containers to ensure the environment is in its expected state.

The only work to do for this issue that I can see (using actual python modules for the --event_handler sync script option) was captured in #199... and that issue was closed as resolved/invalid. So, we will close this the same way. We can always make a new issue if this idea becomes more interesting in the future.