fulcrumgenomics / fgoxide

Quality of life improvements in Rust.
MIT License
5 stars 3 forks source link

Add new_writer and new_reader to DelimFile #13

Closed nh13 closed 1 year ago

nh13 commented 1 year ago

I want to write out the delimited data as I generate each row/line for the case I have a very long and wide tab-delimited file.

jdidion commented 1 year ago

@nh13 To make sure I understand, you want to have the following:

fn new_reader<D: Deserialize, P: AsRef<Path>>(
  &self,
  path: P,
  delimiter: u8,
  quote: bool,
) -> Result<DelimFileReader<D>> { ... }

fn new_writer<S: Serialize, P: AsRef<Path>>(
  &self,
  path: P,
  delimiter: u8,
  quote: bool,
) -> Result<DelimFileWriter<S>> { ... }

where DelimFileReader has a read() -> D method and DelimFileWriter has a write(record: S) method.

It probably also makes sense to re-write the other DelimFile functions using the reader/writer.

nh13 commented 1 year ago

@jdidion I think those are the two methods I want, so I can lazily read from a very large delimited file, and lazily write each record (as I create) it. One could also have two convenience methods: iterator that returns an iterator over the records (Iterator<Result<D>>) when reading, and a method to write records from an iterator (Iterator<S>) to a path. Thoughts?

I also agree that when you have these methods, the rest of DelimFile can be re-written.