FileNotFoundError: [Errno 2] No such file or directory: 'path\\to\\data\\masks_overlayed_eroded_0_dilated_0\\train\\masks\\000000142365'

[QJvic]

I use this in WIN10 ,anaconda. When I typed python main.py -- train --pipeline_name unet_weighted，I got errors as follows.It seems like can't find the png file.But I search it in folder and find 000000142365.png does actually exist. Wish for your replay.

(py36) E:\building\open-solution-mapping-challenge-master>python main.py -- train --pipeline_name unet_weighted
neptune: Executing in Offline Mode.
neptune: Executing in Offline Mode.
2018-10-27 10-57-21 mapping-challenge >>> training
neptune: Executing in Offline Mode.
neptune: Executing in Offline Mode.
2018-10-27 10-57-26 steps >>> step xy_train adapting inputs
2018-10-27 10-57-26 steps >>> step xy_train transforming...
2018-10-27 10-57-26 steps >>> step xy_inference adapting inputs
2018-10-27 10-57-26 steps >>> step xy_inference transforming...
2018-10-27 10-57-26 steps >>> step loader adapting inputs
2018-10-27 10-57-26 steps >>> step loader transforming...
2018-10-27 10-57-26 steps >>> step unet unpacking inputs
2018-10-27 10-57-26 steps >>> step unet fitting and transforming...
2018-10-27 10-57-29 steps >>> starting training...
2018-10-27 10-57-29 steps >>> initial lr: 0.0005
2018-10-27 10-57-29 steps >>> epoch 0 ...
neptune: Executing in Offline Mode.
neptune: Executing in Offline Mode.
neptune: Executing in Offline Mode.
neptune: Executing in Offline Mode.
neptune: Executing in Offline Mode.
neptune: Executing in Offline Mode.
neptune: Executing in Offline Mode.
neptune: Executing in Offline Mode.
Traceback (most recent call last):
  File "main.py", line 93, in <module>
    main()
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "main.py", line 31, in train
    pipeline_manager.train(pipeline_name, dev_mode)
  File "E:\building\open-solution-mapping-challenge-master\src\pipeline_manager.py", line 32, in train
    train(pipeline_name, dev_mode, self.logger, self.params, self.seed)
  File "E:\building\open-solution-mapping-challenge-master\src\pipeline_manager.py", line 116, in train
    pipeline.fit_transform(data)
  File "E:\building\open-solution-mapping-challenge-master\src\steps\base.py", line 106, in fit_transform
    step_inputs[input_step.name] = input_step.fit_transform(data)
  File "E:\building\open-solution-mapping-challenge-master\src\steps\base.py", line 106, in fit_transform
    step_inputs[input_step.name] = input_step.fit_transform(data)
  File "E:\building\open-solution-mapping-challenge-master\src\steps\base.py", line 106, in fit_transform
    step_inputs[input_step.name] = input_step.fit_transform(data)
  [Previous line repeated 3 more times]
  File "E:\building\open-solution-mapping-challenge-master\src\steps\base.py", line 112, in fit_transform
    return self._cached_fit_transform(step_inputs)
  File "E:\building\open-solution-mapping-challenge-master\src\steps\base.py", line 123, in _cached_fit_transform
    step_output_data = self.transformer.fit_transform(**step_inputs)
  File "E:\building\open-solution-mapping-challenge-master\src\steps\base.py", line 262, in fit_transform
    self.fit(*args, **kwargs)
  File "E:\building\open-solution-mapping-challenge-master\src\models.py", line 76, in fit
    for batch_id, data in enumerate(batch_gen):
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\torch\utils\data\dataloader.py", line 336, in __next__
    return self._process_next_batch(batch)
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\torch\utils\data\dataloader.py", line 357, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
FileNotFoundError: Traceback (most recent call last):
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\torch\utils\data\dataloader.py", line 106, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\torch\utils\data\dataloader.py", line 106, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "E:\building\open-solution-mapping-challenge-master\src\loaders.py", line 150, in __getitem__
    Di = self.load_joblib(distance_filepath)
  File "E:\building\open-solution-mapping-challenge-master\src\loaders.py", line 135, in load_joblib
    return joblib.load(filepath)
  File "C:\Users\lenovo\AppData\Local\conda\conda\envs\py36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 590, in load
    with open(filename, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'path\\to\\data\\masks_overlayed_eroded_0_dilated_0\\train\\masks\\000000142365'

[QJvic]

And my pytorch is 0.4 , does this have influence?

[jakubczakon]

Hi @QJvic You should downgrade to pytorch 0.3.1 but it seems that there is something wrong with the paths.

What are your data paths in the neptune.yaml?

[carbonox-infernox]

Your masks directory is probably named 'masks_overlayed_eroded_0_dilated_2', whereas the model is looking for 'masks_overlayed_eroded_0_dilated_0'.

you need to go into src/pipeline_config and edit line 13 to say Y_COLUMNS = ['file_path_mask_eroded_0_dilated_2']

[xdsonglinliu]

@QJvic @carbonox-infernox return joblib.load(filepath) File "C:\Users\MSIK\Anaconda3\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 590, in load with open(filename, 'rb') as f: FileNotFoundError: [Errno 2] No such file or directory: 'D:\\crowdAI\\test0118\\masks_overlayed_eroded_0_dilated_2\\train\\masks\\000000142365'

I had the same problem. My platform is : Win10+anaconda, Python3.7 and Pytorch1.0.

The following code executes correctly. python main.py -- prepare_masks python main.py -- prepare_metadata --train_data --valid_data --test_data It seems that my path settings are correct, but there is no file suffix. Like"000000142365.png"

I hope you can help me. Thanks!

[jakubczakon]

Hi there @xdsonglinliu.

Can you check that this file exists and you can open it separately? For example go to the notebook and run

from sklearn.externals import joblib
joblib.load(TYPE_FILEPATH)

It is also possible, that there are some assumptions about path parsing that we made because we developed the codebase entirely on linux.

[xdsonglinliu]

@jakubczakon Thanks a lot. With your help, the problem has been solved.

[cvKDean]

@xdsonglinliu @jakubczakon Hi, I am also encountering his problem as well. However, I noticed that the file 000000142365.png does not exist in the masks directory after generating the masks through python main.py -- prepare_masks. The difference is I used annotation-small.json instead of annotation.json, since my machine couldn't handle the large amount of files, similar to the situation in #166. This might be the cause why the particular 000000142365.png file was not created.

Is it necessary to use the larger annotation.json file to proceed with training? Thank you very much.

[jakubczakon]

@cvKDean I am afraid something went wrong and you need to rerun with the full annotation.json file. This file is simply not a part of the annotation-small.json.

[cvKDean]

@jakubczakon Thank you for your response. I ran python main.py -- prepare_masks again, this time with annotation.json. And I confirmed that the file 000000142365.png was present in the directory. However, I still encounter the same problem. It seems that loaders.py has the wrong file path (no .png suffix).

I also checked the generated metadata . csv file via python main.py -- prepare_metadata --train_data --valid_data --test_data and I also confirmed that the filepaths are correct, and they have .png suffixes. Am I missing something? Thank you for your patience.

Edit: I am trying to see where the code reads the stage1_metadata.csv file and passes the filepath of the image to load_image in loaders.py, but I can't see exactly where img_filepath is being extracted from.

[angeloprop]

@xdsonglinliu
Can you explain how to fix the problem Please!

[SalmanQasim]

@jakubczakon can you please tell about this error: FileNotFoundError: [Errno 2] No such file or directory: 'data/meta_data/masks_overlayed_eroded_0_dilated_0/train/masks/000000153612.png'

I have already prepared masks from python main.py prepare_masks python main.py prepare_metadata --train_data --valid_data and has generated a metadata.csv file