Crash caused by pandas TextFileReader

[calmacx]

Summary

When turning on chunking (-nc) of data (or using the -np command) there is now a crash...

Crash report

2022-05-18 10:42:47 - LocalDataCollection - INFO - resetting used bricks
Traceback (most recent call last):
  File "/Users/calummacdonald/.pyenv/versions/3.8.0/bin/coconnect", line 33, in <module>
    sys.exit(load_entry_point('co-connect-tools', 'console_scripts', 'coconnect')())
  File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/calummacdonald/.pyenv/versions/3.8.0/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/calummacdonald/Usher/CO-CONNECT/Software/temp/co-connect-tools/coconnect/cli/subcommands/run.py", line 584, in map
    cdm.process(conserve_memory=True)
  File "/Users/calummacdonald/Usher/CO-CONNECT/Software/temp/co-connect-tools/coconnect/cdm/model.py", line 707, in process
    self.inputs.reset()
  File "/Users/calummacdonald/Usher/CO-CONNECT/Software/temp/co-connect-tools/coconnect/io/common.py", line 101, in reset
    brick.reset()
  File "/Users/calummacdonald/Usher/CO-CONNECT/Software/temp/co-connect-tools/coconnect/io/common.py", line 133, in reset
    f = self.__df_handler.f
AttributeError: 'TextFileReader' object has no attribute 'f'

How to reproduce

environment

$ python -c 'import pandas,sys; print (sys.version); print("pandas",pandas.__version__)'
3.8.0 (default, Mar 11 2021, 10:43:06) 
[Clang 12.0.0 (clang-1200.0.32.29)]
pandas 1.4.2

coconnect run map --rules rules.json data/* -np 100 -nc 50  --allow-missing-data

• run the transform tool with any chunking (-nc) or maximising the number of rows to chink

Suspected cause of the problem

• I update my version of pandas to 1.4.2 from 1.3.X
  • https://github.com/pandas-dev/pandas/blob/1.4.x/pandas/io/parsers/readers.py
  • https://github.com/pandas-dev/pandas/blob/1.3.x/pandas/io/parsers/readers.py#L774
• pandas removed the attribute causing this crash

What we need to do

• [ ] find a way to retrieve the original file name (was self.f in 1.3.X) from the TextFileReader / df
• [ ] force the version of pandas to be stable in requirements.txt