search

ddialliance / sdtl

Other
1 stars 0 forks source link

SDTH should file be abstracted? #3

Open DanSmith opened 2 months ago

DanSmith commented 2 months ago

Database, endpoint, etc are all similar concepts. Should file be more generalized?

altergc commented 2 months ago

For background, this question arose from a discussion with colleagues at Making-Sense (https://making-sense.info/), who are looking at using SDTH to describe the operations in a VTL (https://sdmx.org/?page_id=5096) script. VTL distinguishes between persistent and transient datasets. However, VTL considers a dataset persistent if it is transferred from one VTL script to another. Since this transfer could happen entirely in silica, a VTL persistent dataset may never exist on a permanent storage device.

SDTH assumes that a FileInstance is not in in computer memory, because it must be Loaded to be used and Saved. In contrast, a DataframeInstance is transient, because it will disappear when execution of the script ends.

"File" implies a storage device. Should we use a more general concept that is more consistent with VTL?