Error of -- indoor3d_sem_seg_hdf5_data_Area_5_1.0m_0.5s_train --

[yunusbaskann]

INDOOR3D_DATA_DIR/Area_1_hallway_6.npy (474, 4096, 9), (474, 4096) INDOOR3D_DATA_DIR/Area_1_hallway_7.npy (390, 4096, 9), (390, 4096) Traceback (most recent call last): File "gen_indoor3d_h5.py", line 125, in insert_batch(data, label, i == len(data_label_files)-1) File "gen_indoor3d_h5.py", line 90, in insert_batch data_prep_util.save_h5(h5_filename, h5_batch_data, h5_batch_label, data_dtype, label_dtype) File "/home/yunus/Downloads/RSNet-master/data/utils/data_prep_util.py", line 80, in save_h5 h5_fout = h5py.File(h5_filename) File "/home/yunus/anaconda3/lib/python3.8/site-packages/h5py/_hl/files.py", line 424, in init fid = make_fid(name, mode, userblock_size, File "/home/yunus/anaconda3/lib/python3.8/site-packages/h5py/_hl/files.py", line 190, in make_fid fid = h5f.open(name, flags, fapl=fapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 96, in h5py.h5f.open OSError: Unable to open file (unable to open file: name = './indoor3d_sem_seg_hdf5_data_Area_5_1.0m_0.5s_train/ply_data_all_0.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

[MCemGunes]

Hey @yunusbaskann, I've been dealing with the exact same problem. I searched the solutions for the h5py file path errors but none of them worked or I could not manage to apply them properly. Here are my terminal outputs, I would be glad if you can share your solution.

`root@021f5532d2f1:/app/RSNet/data# python3 gen_indoor3d_h5.py --indoor3d_data_dir /app/RSNet/collected/ --split train 204/272 files selected

/app/RSNet/collected/Area_1_conferenceRoom_1.npy (80, 4096, 9), (80, 4096) /app/RSNet/collected/Area_1_conferenceRoom_2.npy (132, 4096, 9), (132, 4096) /app/RSNet/collected/Area_1_copyRoom_1.npy (35, 4096, 9), (35, 4096) /app/RSNet/collected/Area_1_hallway_1.npy (32, 4096, 9), (32, 4096) /app/RSNet/collected/Area_1_hallway_2.npy (42, 4096, 9), (42, 4096) /app/RSNet/collected/Area_1_hallway_3.npy (32, 4096, 9), (32, 4096) /app/RSNet/collected/Area_1_hallway_4.npy (25, 4096, 9), (25, 4096) /app/RSNet/collected/Area_1_hallway_5.npy (32, 4096, 9), (32, 4096) /app/RSNet/collected/Area_1_hallway_6.npy (474, 4096, 9), (474, 4096) /app/RSNet/collected/Area_1_hallway_7.npy (390, 4096, 9), (390, 4096) Traceback (most recent call last): File "gen_indoor3d_h5.py", line 126, in insert_batch(data, label, i == len(data_label_files)-1) File "gen_indoor3d_h5.py", line 90, in insert_batch data_prep_util.save_h5(h5_filename, h5_batch_data, h5_batch_label, data_dtype, label_dtype) File "/app/RSNet/data/utils/data_prep_util.py", line 80, in save_h5 h5_fout = h5py.File(h5_filename) File "/opt/conda/lib/python3.6/site-packages/h5py/_hl/files.py", line 427, in init swmr=swmr) File "/opt/conda/lib/python3.6/site-packages/h5py/_hl/files.py", line 190, in make_fid fid = h5f.open(name, flags, fapl=fapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 96, in h5py.h5f.open OSError: Unable to open file (unable to open file: name = '/app/RSNet/data/processed/indoor3d_sem_seg_hdf5_data_Area_5_1.0m_0.5s_train/ply_data_all_0.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)`

I updated get_indoor3d_h5.py as seen below: `parser.add_argument('--data_dir', default='/app/RSNet/data/processed', help='directory to hold output data')

parser.add_argument('--indoor3d_data_dir', default='/app/RSNet/collected/', help='directory of processed stanford indoor3d dataset')`

[MCemGunes]

Hello again, after a short session of debugging I figured out there is a problem with the definition of the
h5_fout = h5py.File(h5_filename) method in the data_prep_utils.py in line 80. The command expects you to give the read or write status of the directory. So we can create if it does not exist and skip if it already exists (You can choose the proper option as you wish. Resource: h5py Documentation ) To sum up you should update that line as is shown below: h5_fout = h5py.File(h5_filename,'w-')

---

Solution Ended Here After This Part, I'm adding the code used to debug the problem.

gen_indoor3d_h5.py ==> updated insert_batch method Prints the sizes of the batches and each file

` def insert_batch(data, label, last_batch=False):

global h5_batch_data, h5_batch_label
global buffer_size, h5_index
data_size = data.shape[0]
print('data size: ', data_size)
print('batch size: ', buffer_size) 
# If there is enough space, just insert
if buffer_size + data_size <= h5_batch_data.shape[0]:
    print('h5 batch shape: ',h5_batch_data.shape[0])
    h5_batch_data[buffer_size:buffer_size+data_size, ...] = data
    h5_batch_label[buffer_size:buffer_size+data_size] = label
    buffer_size += data_size
else: # not enough space
    print('capacity is not enough ')
    capacity = h5_batch_data.shape[0] - buffer_size
    print('capacity is ', capacity)
    assert(capacity>=0)
    if capacity > 0:
       print('capacity>0 save the batch and reset buffer size')
       h5_batch_data[buffer_size:buffer_size+capacity, ...] = data[0:capacity, ...] 
       h5_batch_label[buffer_size:buffer_size+capacity, ...] = label[0:capacity, ...] 
    # Save batch data and label to h5 file, reset buffer_size
    print('reshaped_h5 batch_size: ',h5_batch_data.shape[0])

    h5_filename =  output_filename_prefix + '_' + str(h5_index) + '.h5'
    print('h5 batch_filename: ',h5_filename)
    data_prep_util.save_h5(h5_filename, h5_batch_data, h5_batch_label, data_dtype, label_dtype) 
    print('Stored {0} with size {1}'.format(h5_filename, h5_batch_data.shape[0]))
    h5_index += 1
    buffer_size = 0
    # recursive call
    insert_batch(data[capacity:, ...], label[capacity:, ...], last_batch)

if last_batch and buffer_size > 0:
    h5_filename =  output_filename_prefix + '_' + str(h5_index) + '.h5'
    data_prep_util.save_h5(h5_filename, h5_batch_data[0:buffer_size, ...], h5_batch_label[0:buffer_size, ...], data_dtype, label_dtype)
    print('Stored {0} with size {1}'.format(h5_filename, buffer_size))
    h5_index += 1
    buffer_size = 0
return

` data/utils/data_prep_util ==> updated save_h5, The method that saves the .h5 outputs.

` def save_h5(h5_filename, data, label, data_dtype='uint8', label_dtype='uint8'):

print('h5_filename is ', h5_filename)
h5_fout = h5py.File(h5_filename,'w-')
print('saving code is called..')
h5_fout.create_dataset(
        'data', data=data,
        compression='gzip', compression_opts=4,
        dtype=data_dtype)
h5_fout.create_dataset(
        'label', data=label,
        compression='gzip', compression_opts=1,
        dtype=label_dtype)
h5_fout.close()`

Last outputs of the code:

/home/ubuntuoslo/git_repo/docker_repos/data_deneme/RSNet/RSNet/data/Area_6_office_9.npy (132, 4096, 9), (132, 4096) sample couunt data size: 132 batch size: 356 h5 batch shape: 500 /home/ubuntuoslo/git_repo/docker_repos/data_deneme/RSNet/RSNet/data/Area_6_openspace_1.npy (210, 4096, 9), (210, 4096) sample couunt data size: 210 batch size: 488 capacity is not enough capacity is 12 capacity>0 save the batch and reset buffer size reshaped_h5 batch_size: 500 h5 batch_filename: /home/ubuntuoslo/git_repo/docker_repos/data_deneme/RSNet/RSNet/data/indoor3d_sem_seg_hdf5_data_Area_5_1.0m_0.5s_train/ply_data_all_32.h5 h5_filename is /home/ubuntuoslo/git_repo/docker_repos/data_deneme/RSNet/RSNet/data/indoor3d_sem_seg_hdf5_data_Area_5_1.0m_0.5s_train/ply_data_all_32.h5 saving code is called.. Stored /home/ubuntuoslo/git_repo/docker_repos/data_deneme/RSNet/RSNet/data/indoor3d_sem_seg_hdf5_data_Area_5_1.0m_0.5s_train/ply_data_all_32.h5 with size 500 data size: 198 batch size: 0 h5 batch shape: 500 /home/ubuntuoslo/git_repo/docker_repos/data_deneme/RSNet/RSNet/data/Area_6_pantry_1.npy (35, 4096, 9), (35, 4096) sample couunt data size: 35 batch size: 198 h5 batch shape: 500 h5_filename is /home/ubuntuoslo/git_repo/docker_repos/data_deneme/RSNet/RSNet/data/indoor3d_sem_seg_hdf5_data_Area_5_1.0m_0.5s_train/ply_data_all_33.h5 saving code is called.. Stored /home/ubuntuoslo/git_repo/docker_repos/data_deneme/RSNet/RSNet/data/indoor3d_sem_seg_hdf5_data_Area_5_1.0m_0.5s_train/ply_data_all_33.h5 with size 233 Total samples: 16733