Error when running reconcile on csv files

[rbruhn]

We are currently running an older version of label_reconciliations on Biospex. In exploring whether to update the server or not, I was testing the current tag v0.6.0 on Ubuntu 20.04 in a local environment. I installed Python 3.10.13 and created the virtual environment successfully. However, when running the script, it's throwing an error.

Running code:

venv/bin/python ./reconcile.py --reconciled data/reconciled.csv --summary data/summary.html data/314.csv

The csv file is simply the normal classification data we download from Notes From Nature. 6086 records. There are no duplicate columns.

classification_id | user_name | user_id | user_ip | workflow_id | workflow_name | workflow_version | created_at | gold_standard | expert | metadata | annotations | subject_data | subject_ids

A reconcile.csv file with 2000 rows is created and appears to be in order.

Any idea what might be causing this?

venviro/bin/python ./reconcile.py --reconciled data/reconciled.csv --summary data/summary.html data/314.csv
/home/vagrant/code/server/reconciliations/pylib/table.py:70: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
  df[columns] = df[columns].applymap(self.get_result)
/home/vagrant/code/server/reconciliations/pylib/table.py:61: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
  df[columns] = df[columns].applymap(self.get_note)
/home/vagrant/code/server/reconciliations/pylib/summary.py:92: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
  df = df.applymap(create_link)
/home/vagrant/code/server/reconciliations/pylib/summary.py:292: FutureWarning:

Styler.applymap_index has been deprecated. Use Styler.map_index instead.

Traceback (most recent call last):
  File "/home/vagrant/code/server/reconciliations/./reconcile.py", line 229, in <module>
    main()
  File "/home/vagrant/code/server/reconciliations/./reconcile.py", line 222, in main
    summary.report(args, unreconciled, reconciled)
  File "/home/vagrant/code/server/reconciliations/pylib/summary.py", line 43, in report
    results=get_results(reconciled, problems),
  File "/home/vagrant/code/server/reconciliations/pylib/summary.py", line 294, in get_results
    return style.to_html()
  File "/home/vagrant/code/server/reconciliations/venviro/lib/python3.10/site-packages/pandas/io/formats/style.py", line 1348, in to_html
    html = obj._render_html(
  File "/home/vagrant/code/server/reconciliations/venviro/lib/python3.10/site-packages/pandas/io/formats/style_render.py", line 204, in _render_html
    d = self._render(sparse_index, sparse_columns, max_rows, max_cols, "&nbsp;")
  File "/home/vagrant/code/server/reconciliations/venviro/lib/python3.10/site-packages/pandas/io/formats/style_render.py", line 161, in _render
    self._compute()
  File "/home/vagrant/code/server/reconciliations/venviro/lib/python3.10/site-packages/pandas/io/formats/style_render.py", line 256, in _compute
    r = func(self)(*args, **kwargs)
  File "/home/vagrant/code/server/reconciliations/venviro/lib/python3.10/site-packages/pandas/io/formats/style.py", line 1755, in _apply
    self._update_ctx(result)
  File "/home/vagrant/code/server/reconciliations/venviro/lib/python3.10/site-packages/pandas/io/formats/style.py", line 1552, in _update_ctx
    raise KeyError(
KeyError: '`Styler.apply` and `.map` are not compatible with non-unique index or columns.'

[PmasonFF]

Since version 0.6.0 was released pandas has undergone a major revision from 1.5.X to 2.1.3 as of this writing. You could try to install an older version of pandas for now, or you could try the latest development version of the whole reconcile.py suite from the git (0.8.4 as of this writing) with the current version of pandas. This later course is perhaps less tested, but at least you should be current with some working versions of the dependencies.

I am still running Python 3.10.2, reconcile.py 0.6.6 with pandas 1.5.3 with no problems but by that time table.py and summary.py had already been revised to no longer use .applymap. If you really get stuck I could send you a clone of what I have, and the versions of all the dependencies I am running.

Pmason

[rbruhn]

@PmasonFF Thank you for the information. I checked v0.6.0 and altered the requirement.txt to use pandas 1.5.3 and it worked perfectly. Closing the issue.