Closed SteveBronder closed 5 months ago
managing the files is a PITA, but:
that said, if we move to Apache Arrow / more granular outputs, it would make sense because otherwise, gazillions of files.
As long as we had a column for chain IDs I'd the file we could still compute rhat.
Did we intend to have an option that if a chain is slow we kill it? Idk how this would stop that since we'd just stop writing that chain to the file
is the problem here managing the output files in CmdStanR?
No it's just in general this feels like a nice format for reading upstream in general. Instead of having to read multiple files and combine them upstream things would just need to read one file.
As the number of chains grows I could see you having issues with contending for the lock on the file handle
Yeah I agree with the contention issue. We would probably have to do something kind of silly with mmap and an offset to get reasonable speeds. We'd also have to change how we write out the mass matrix since we would have many for one file.
I'm going to close this as "probably not worth it"
Summary:
Right now we write each chain to a separate file. Should we just write all chains to one file and have a
chain
column to say what chain the sample came from? It would add a new column to parse which could hurt backwards compatability for our output. But I think it would end up being making reading and writing a little simplerAdditional Information:
Provide any additional information here.
Current Version:
v2.34.1