I was experimenting with Beam Dataframe API through Beam Notebooks + Interactive runner and wasn't able to use fillna on individual columns. Here is a repro on a dataframe with two columns:
%%writefile numbers.csv
col1,col2
1,1
NaN,1
-1,1
import apache_beam as beam
import apache_beam.runners.interactive.interactive_beam as ib
from apache_beam.runners.interactive.interactive_runner import InteractiveRunner
pipeline = beam.Pipeline(InteractiveRunner())
beam_df = pipeline | 'Read CSV' >> beam.dataframe.io.read_csv('numbers.csv')
beam_df['col1'] = beam_df['col1'].fillna(0)
ib.collect(beam_df)
This fails with ValueError: "[ConstantExpression[constant_int_140226804834624]]:140226804986784" requires a pipeline to be specified as there are no deferred inputs.
What happened?
I was experimenting with Beam Dataframe API through Beam Notebooks + Interactive runner and wasn't able to use
fillna
on individual columns. Here is a repro on a dataframe with two columns:This fails with ValueError: "[ConstantExpression[constant_int_140226804834624]]:140226804986784" requires a pipeline to be specified as there are no deferred inputs.
A rewritten version:
also fails.
The snippets pass without issues on Pandas or Dask.
Full Stacktrace
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[62], line 16 14 beam_df = pipeline | 'Read CSV' >> beam.dataframe.io.read_csv('numbers.csv') 15 beam_df['col1'] = beam_df['col1'].fillna(0) ---> 16 ib.collect(beam_df) # fails with ValueError: "[ConstantExpression[constant_int_140226804834624]]:140226804986784" requires a pipeline to be specified as there are no deferred inputs. 18 c1 = beam_df['col1'] 19 c1 = c1.fillna(0) File /jupyter/.kernels/apache-beam-2.56.0/packages/beam/sdks/python/apache_beam/runners/interactive/utils.py:277, in progress_indicated.Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components