$'\r': command not found

[VwakeM]

I'm running the docker image for camera-trap-classifier from https://hub.docker.com/r/will5448/camera-trap-classifier on a computing cluster. The create_dataset_inventorycommand gives the following error:

./condor_exec.exe: line 2: $'\r': command not found
./condor_exec.exe: line 7: $'\r': command not found
./condor_exec.exe: line 9: -export_path: command not found
./condor_exec.exe: line 10: $'\r': command not found
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/config.py", line 558, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/lib/python3.5/logging/config.py", line 731, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.5/dist-packages/camera_trap_classifier/config/logging.py", line 16, in logmaker
    return logging.FileHandler(log_path, mode, encoding)
  File "/usr/lib/python3.5/logging/__init__.py", line 1008, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python3.5/logging/__init__.py", line 1037, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/home/malleshappa/dataset_inventory/run_debug.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/ctc.create_dataset_inventory", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/camera_trap_classifier/create_dataset_inventory.py", line 136, in main
    setup_logging(log_output_path=args['log_outdir'])
  File "/usr/local/lib/python3.5/dist-packages/camera_trap_classifier/config/logging.py", line 35, in setup_logging
    logging.config.dictConfig(config)
  File "/usr/lib/python3.5/logging/config.py", line 795, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/lib/python3.5/logging/config.py", line 566, in configure
    '%r: %s' % (name, e))
ValueError: Unable to configure handler 'debug_file_handler': [Errno 2] No such file or directory: '/home/malleshappa/dataset_inventory/run_debug.log'

condor_exec.exe is the scheduling program used by the computing cluster.

[marco-willi]

These messages seem to indicate the command line arguments have not been passed correctly to 'create_dataset_inventory' and/or contain special characters:

./condor_exec.exe: line 2: $'\r': command not found
./condor_exec.exe: line 7: $'\r': command not found
./condor_exec.exe: line 9: -export_path: command not found
./condor_exec.exe: line 10: $'\r': command not found

How are you calling 'create_dataset_inventory'?

What is printed to the terminal? Are the arguments what you expect them to be?

Does ctc.create_dataset_inventory --help work?

Have you mapped paths from inside the container to outside the container? If that path does not exist inside the container: '/home/malleshappa/dataset_inventory/run_debug.log' the code would crash. If you use Docker you can map paths using something like this: -v /home/malleshappa/dataset_inventory/:/data/ when you start the container.

[VwakeM]

Thanks, Marco. You are right - my directories are outside the container. My modified command looks like this: ctc.create_dataset_inventory dir -path -v /home/malleshappa/images/:/data/ -export_path -v /home/malleshappa/dataset_inventory/:/data/

I still get the $'\r': command error. Logs below:

./condor_exec.exe: line 2: $'\r': command not found ./condor_exec.exe: line 7: $'\r': command not found ./condor_exec.exe: line 9: $'\r': command not found ./condor_exec.exe: line 11: $'\r': command not found usage: CREATE DATSET INVENTORY dir [-h] -path PATH -export_path EXPORT_PATH CREATE DATSET INVENTORY dir: error: argument -path: expected one argument ./condor_exec.exe: line 13: $'\r': command not found

[marco-willi]

Sorry to hear you are still having issues. These messages seem to indicate problems with how you try to run the code and not the code itself. Nevertheless, let me try to help if I can:

1. The $'\r' message indicates you have special characters, e.g. line breaks, in your command. This is always problematic on unix systems. You can try to get rid of them or build your command on one line without any line breaks.

2. Generally, if you try to use Docker, the command probably looks like this:

   sudo docker run --name ctc -v /my_data/:/data/ -itd camera-trap-classifier:latest-cpu
   sudo docker exec ctc ctc.create_dataset -inventory /data/dataset_inventory.json \
   -output_dir /data/tfr_files/ \
   -image_save_side_smallest 200 \
   -split_percent 0.5 0.25 0.25 \
   -overwrite

Note that you need to map paths only once! And then use these mapped paths for the parameters.

3. If you don't have 'sudo' rights on your cluster you may not be able to use Docker. In that case you may be able to user 'singularity', for example: Singularity

4. If neither Docker or Singularity is available on your cluster, you need to run the command like you did on your local machine:

   ctc.train ....

5. If nothing works I'd consult with the people who manage your cluster. I'm not familiar with condoer_exec.exe nor how you are supposed to run Docker containers on your cluster. A simple test could be to try to run the hello-world Docker example to rule out issues with Docker: Docker Get Started

[VwakeM]

Thanks, Marco. I was confused as the stack trace had two exceptions and I thought they were both the same. I got rid of the special characters and $'\r': command error has gone away. I don't have 'sudo' rights on the cluster, but there seems to be a cluster specific workaround for it. So, hopefully that should resolve all issues and I can train my model.