Closed johncoxon closed 2 years ago
Thanks for reporting this! If I switch the engine to pyarrow
on the to_parquet()
step, the problem is resolved. The read_parquet()
can be either fastparquet
or pyarrow
. I'm not sure why fastparquet is doing this, we'll need to investigate more, but at least that should get you unblocked!
I have repeated your exact code (assuming filename = "data/accounts*.csv"
), and encountered no problem. What version of fastparquet do you have?
Do the files written to the target directory have any size? Is there any error or warning?
storage_options={"has_nulls": True},
This does not look like a storage/filesystem option, but something to pass to the parquet writer; also, it should not be necessary, all columns are assumed to be nullable.
@martindurant I also found the same thing,and got no errors or warnings... However, in the context of the tutorial, the next cell suggests implementing some code that requires the DataFrame not be empty, which I suspect led to this issue.
Thanks both! The parquet directory is three 25MB files in part.0.parquet
and in part 1 and 2, so 75MB total.
We have removed this chapter so I'll go ahead and close this issue.
The code in the seventh chapter does not work to read the data back in from parquet after it's been written to disk.
What happened: The attempt to read the data back in from the parquet files fails.
What you expected to happen: The attempt to succeed so I could continue with the chapter examples.
MWE: (I've written this to illustrate the issue in a single block, but for me just following the cells in 07 reproduces the issue, too.)
yields the following output:
Environment: