datajoint / datajoint-matlab

Relational data pipelines for the science lab
MIT License
42 stars 38 forks source link

Occasional fwrite errors using file-based stores #381

Open zfj1 opened 3 years ago

zfj1 commented 3 years ago

Bug Report

Description

Inserting into a store using the file protocol occasionally causes an error in fwrite.

Reproducibility

Error in dj.store_plugins.File.upload_buffer (line 57) fwrite(fileID, buffer);

Error in dj.internal.ExternalTable/upload_buffer (line 116) self.spec.upload_buffer(packed_cell{1}, self.make_uuid_path(uuid, ''));

Error in dj.Relvar/insert/makePlaceholder (line 420) value = self.schema.external.table(...

Error in dj.Relvar/insert (line 499) [v, placeholder] = makePlaceholder(i, tuple.(header.attributes(i).name));


### Expected Behavior
Successful insertion

### Screenshots
N/A

### Additional Research and Context
This often happens in the middle of inserting many entries into a store-based table, causing the transaction to fail. It then will often work after re-running the insertion.

Our store is on a mounted network drive, so it's possible that momentary disconnections from the file server are at play, but I would think this is probably a common use case. (Note that our file server and db server are on the same network). A simple solution may just entail retrying the file creation on failure.

It would also be convenient to disable the 'directory already exists warning' using `warning('off','MATLAB:MKDIR:DirectoryExists')`