All cloud behavior right now goes through a two stage process:
1) serialize to local storage
2) upload
and in reverse when loading
1) download to local storage
2) deserialize
Effectively that uses an implicit disk_{} save pattern for each object even if that is undesirable.
Proposal:
1) serialize objects to a tempfiles location
2) copy to save locations (copy to local for disk patterns and upload directly for cloud)
in reverse:
1) download to tempfiles
2) deserialize
os should flush the temp storage as needed leaving the system as expected. potential downside is the loss of cache behavior that could potentially be mitigated by a named temp directory for consistent access
eyadgaran
commented
2 years ago
All cloud behavior right now goes through a two stage process: 1) serialize to local storage 2) upload
and in reverse when loading 1) download to local storage 2) deserialize
Effectively that uses an implicit disk_{} save pattern for each object even if that is undesirable.
Proposal: 1) serialize objects to a tempfiles location 2) copy to save locations (copy to local for disk patterns and upload directly for cloud)
in reverse: 1) download to tempfiles 2) deserialize
os should flush the temp storage as needed leaving the system as expected. potential downside is the loss of cache behavior that could potentially be mitigated by a named temp directory for consistent access