xitongsys / parquet-go

pure golang library for reading/writing parquet file
Apache License 2.0
1.25k stars 294 forks source link

Need help in reading parquet file in chunks from GCS #578

Open saiteja2021 opened 7 months ago

saiteja2021 commented 7 months ago

using multi Reader to read the file in chunks

multiReader := io.MultiReader(mReader)

    for {
        buf := make([]byte, 1024)
        n, err := multiReader.Read(buf)
        if err != nil {
            if err == io.EOF {
                break
            }
            return nil, err
        }

        pr, err := pr.NewParquetReader(buf, new(ProcLog), 4)
        if err != nil {

            return nil, err
        }
        fmt.Println(string(buf[:n]))
    }

now wants to convert only one chunk to string from parquet format. like converting parquet data in buffer to string or a struct Any help is highly appreciated .

hangxie commented 5 days ago

Never tried myself but https://github.com/xitongsys/parquet-go-source/blob/master/buffer/buffer.go#L26 sounds the one you are looking for.