When working on masking incubator with Claudio, we found a bug in FitsReader.read() implementation. When FitsReader is constructed with column_names/skip_columns_names and used on the mapping stage with read_columns!=None, it does column filtering twice, which leads to an exception complaining about unknown columns
Pseudo-code:
args = ImportArguments(file_reader=FitsReader(skip_column_names=['bad_column']), ...)
pipeline_with_client(args, client) # fails with "column "bad_column" does not exist"
Before submitting
Please check the following:
[x] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
[ ] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
[ ] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
Bug report
When working on masking incubator with Claudio, we found a bug in
FitsReader.read()implementation. WhenFitsReaderis constructed withcolumn_names/skip_columns_namesand used on the mapping stage withread_columns!=None, it does column filtering twice, which leads to an exception complaining about unknown columnsPseudo-code:
Before submitting Please check the following: