the preprocessor now writes processed edge and node data to s3, but the data is split into many files. need to combine them.
the following errors when the files are small,
s3_obj.merge(output_filename, files_list)
throws the error EntityTooSmall.
once we have a single file, we can look into converting that to binary. Alternatively, we can define a custom writer that outputs in binary format without the intermediate csv files.
the preprocessor now writes processed edge and node data to s3, but the data is split into many files. need to combine them.
the following errors when the files are small,
throws the error
EntityTooSmall.once we have a single file, we can look into converting that to binary. Alternatively, we can define a custom writer that outputs in binary format without the intermediate csv files.