What purpose does ArrowRecordBatch solve?

[shivam-880]

I was going through the Flight Java Example and was wondering if we can persist VectorSchemaRoot directly in the Dataset instead of ArrowRecordBatch list?

class Dataset implements AutoCloseable {
    private final List<ArrowRecordBatch> batches;
    private final Schema schema;
    private final long rows;
    public Dataset(List<ArrowRecordBatch> batches, Schema schema, long rows) {
        this.batches = batches;
        this.schema = schema;
        this.rows = rows;
    }
    public List<ArrowRecordBatch> getBatches() {
        return batches;
    }
    public Schema getSchema() {
        return schema;
    }
    public long getRows() {
        return rows;
    }
    @Override
    public void close() throws Exception {
        AutoCloseables.close(batches);
    }
}

[lwhite1]

The record batch is a representation of a RecordBatch IPC message, which is used for Dataset transfer. A VectorSchemaRoot isn't implemented in a way that the conversion process could be skipped.

[lidavidm]

@iamsmkr do you still have questions here?