ransomw1c
commented
3 years ago the proposal and client-side WAL implementation are progress on this iss.
these items do not cover this issue because this issue is envisioning additional golang to not just upload to a context (in the datamon sense, post WAL implementation, not the generalized golang sense, a context mostly characterized by a set of [AWS/gg] buckets) that contains a WAL but also to migrate data between two contexts according to some specialized rules. in general, this would effectively allow end-user functionalities like incremental uploads. i've added the "spike" label to indicate that the exact functionalities for transfer between contexts – much less mapping these 'machine level instructions' (i.e. the scope of the spike) to 'productizeable features' (e.g. incremental uploads).
at time of writing, i opine features such as progress bars and others that can be realized without multiple blob buckets (or multiple contexts, whichever requirement rings truer – they are used interchangeably here) as described #443 are merely papering over the engineering work, something like very simple GC, essentially, described in this issue.
as part of the plans for accelerator pods (to support either search- or query-like functionality, more details anon), we need to allow
~/.datamon/datamon.yamlto ref an additional bucket. this bucket will be used to facet searches, journal writes, etc. by pipeline- or otherwise project-specific in-cloud accelerators.for the scope of this task, we're essentially talking about adding journaling (in the sense of journalled filesystems) or Write Ahead Logging (in the sense of Postgres) to commands that write to existing (meta and blob) stores.