Move space file operations inside package space - Githubissues

brimdata / super

A novel data lake based on super-structured data

https://zed.brimdata.io/

BSD 3-Clause "New" or "Revised" License

1.39k stars 64 forks source link

Move space file operations inside package space #735

Closed alfred-landrum closed 4 years ago

alfred-landrum commented 4 years ago

We need to move the local file operations outside of package ingest, and into package space:

Unexport AllZngFile and the other filenames in space.go
Unexport DataPath
Change the operations in pcap and log ingest so that the work to index or prepare the tmp files happens in package ingest, but that the final act of moving the new zng file or equivalent happens in side package space.

We will need new exported methods to be used by the ingest operations. For example:

func (s *Space) ReplaceAllData(newZngFile string) error {
    // update any cached span info...
return os.Rename(newZngFile, s.dataPath(allZngFile))
}

These new operations will be points for concurrency protection, and eventually handle spaces not backed by local files.

alfred-landrum commented 4 years ago

@mattnibs has some good ideas about how to implement this via a storage interface that a Space would present to operations for ingest & searching.