-
I love your project enabling next-gen OSM analyses with the power of DuckDB. Wouldn't it be cool, to enable specifying a DuckDB file (and optionally table) to store the data in? I know DuckDB has a na…
-
Recently, parquet added support for columnar/modular encryption in version parquet-mr 1.12 ([IBM](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=scripts-parquet-encryption), [GitHub](https:/…
-
https://stackoverflow.com/questions/72340767/control-the-compression-level-when-write-parquet-files-using-polars-in-rust
-
We're trying to write some 14B rows (about 3.6 TB in parquets) to parquet files. When our ETL job finishes, it throws this exception, and the status is "died in job commit".
2015-05-14 09:24:28,158 F…
-
Dear all,
I downloaded the structures from the release [v2023_02 with TM and pLDDT > 70 (71 .tar.gz files)](https://github.com/facebookresearch/esm/blob/main/scripts/atlas/v2023_02/full/tarballs.t…
-
-
A small fraction of the skysim5000 healpixels (52 out of 1568) in hdf5 format have an inconsistent schema for some native quantities which prevent their conversion to parquet format.
The following f…
-
### Problem description
I would like to be able to track dataframe specific metadata through processing, serialization, and deserialization.
A common use case for dataframe metadata is to store da…
-
I have some parquet files which are created by Spark converting AVRO file to parquet file. And these parquet files contain different data types like decimal, int,string. But when I read this file usin…
-
I have an external process that produce Parquet files incrementally, basically it just add files , I want to convert it to Delta Table , which currently works the first time but errors the second time…