xitongsys / parquet-go

pure golang library for reading/writing parquet file
Apache License 2.0
1.27k stars 293 forks source link

GCS filesystem read/write example #436

Closed imalkov82 closed 2 years ago

imalkov82 commented 2 years ago

Can you provide an example of how to use the package to write/read parquet files to Google Cloud Storage

hangxie commented 2 years ago

Use NewGcsFileReader to read:

    gcsReader, err := gcs.NewGcsFileReader(context.Background(), projectId, bucket, objectName))
    if err != nil {
        return nil, fmt.Errorf("failed to open GCS object [%s]: %s", uri, err.Error())
    }
    reader, err := reader.NewParquetReader(gcsReader, nil, 4)
    // everything else is same as dealing with local file.

And NewGcsFileWriter to write.

Assume you have environment configured already to access GCS, if not: https://cloud.google.com/docs/authentication/production#automatically

imalkov82 commented 2 years ago

Awesome, thank you @hangxie !