Index issues in multi-process usage

[oliversalzburg]

I know that this isn't an issue with this module per se, but I thought I'd put it up for discussion anyway.

Without thinking it through properly, I recently integrated this module into one of our worker processes. This process runs clustered. So, when multiple files are written by different workers, their internal states will diverge. Every worker will think the same ID is to be assigned to the next file. Obviously, this results in major conflicts.

The obvious solution is not do do this, of course :D However, I was wonder whether it would be feasible to approach and solve this within this module. I could not come up with a quick, clever solution that wouldn't involve constantly reading back the index database and lockfiles.

One approach I liked was to simply assign UUIDs to files, instead of a simple integer. Then I could use v4 UUIDs and be a lot safer. From what I can tell, this should work, as insert accepts an id parameter. The only problem here is that it is forced to an int.

Please let me know your thoughts and I completely understand if you feel that this is not within the scope of the module.

[grantcarthew]

filestorage is a great package however only designed for a single node. That's why I made scalable-blob-store @oliversalzburg.

It might be what you are after.

[oliversalzburg]

@grantcarthew Awesome! I'll check it out. Thanks for the hint

[petersirka]

@grantcarthew thanks for the reply. This module can't be for the cluster, it's problematic to create.

Thanks!