search

teamtomo / mdocfile

SerialEM mdoc files as pandas dataframes
https://teamtomo.org/mdocfile/
BSD 3-Clause "New" or "Revised" License
9 stars 7 forks source link

Fails to import mdocs in tomo_model #8

Closed EuanPyle closed 1 year ago

EuanPyle commented 1 year ago

Commit 4ce1f2627ba3e99282fec2035bb31cbf81ce4e75 causes the error below. Also, if camel_to_snake is going to be removed, in tomo_model src/tomography_python_programs/import_tilt_series/serialem.py the camel_to_snake=True bit needs to be removed. I won't do that just yet in case camel_to_snake needs to be reintroduced to fix this!

Error in sys.excepthook: Traceback (most recent call last): File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/syntax.py", line 497, in tokens_to_spans _token_type, token = next(tokens) StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/console.py", line 1694, in print extend(render(renderable, render_options)) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/console.py", line 1330, in render yield from self.render(render_output, _options) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/console.py", line 1326, in render for render_output in iter_render: File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/constrain.py", line 29, in rich_console__ yield from console.render(self.renderable, child_options) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/console.py", line 1326, in render for render_output in iter_render: File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/panel.py", line 220, in rich_console lines = console.render_lines(renderable, child_options, style=style) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/console.py", line 1366, in render_lines lines = list( File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/segment.py", line 292, in split_and_crop_lines for segment in segments: File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/console.py", line 1326, in render for render_output in iter_render: File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/padding.py", line 97, in rich_console lines = console.render_lines( File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/console.py", line 1366, in render_lines lines = list( File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/segment.py", line 292, in split_and_crop_lines for segment in segments: File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/console.py", line 1330, in render yield from self.render(render_output, _options) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/console.py", line 1326, in render for render_output in iter_render: File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/syntax.py", line 609, in __rich_console segments = Segments(self._get_syntax(console, options)) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/segment.py", line 668, in init self.segments = list(segments) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/syntax.py", line 637, in _get_syntax text = self.highlight(processed_code, self.line_range) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/syntax.py", line 509, in highlight text.append_tokens(tokens_to_spans()) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/rich/text.py", line 995, in append_tokens for content, style in tokens: RuntimeError: generator raised StopIteration

Original exception was: Traceback (most recent call last): File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc return self._engine.get_loc(casted_key) File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'date_time'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/d/emr207/u/ubcg87b/miniforge/envs/relion/bin/relion_tomo_import", line 8, in sys.exit(cli()) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/typer/main.py", line 328, in call raise e File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/typer/main.py", line 311, in call return get_command(self)(*args, kwargs) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/click/core.py", line 1130, in call return self.main(args, kwargs) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/typer/core.py", line 778, in main return _main( File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/typer/core.py", line 216, in _main rv = self.invoke(ctx) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(args, kwargs) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper return callback(*use_params) # type: ignore File "", line 2, in import_tilt_series_from_serial_em File "/d/emr207/u/ubcg87b/relion-devel-tomo-model/src/tomography_python_programs/_utils/relion.py", line 75, in pipeline_job func(args, **kwargs) File "/d/emr207/u/ubcg87b/relion-devel-tomo-model/src/tomography_python_programs/import_tilt_series/serialem.py", line 128, in import_tilt_series_from_serial_em tilt_image_df = _generate_tilt_image_dataframe( File "/d/emr207/u/ubcg87b/relion-devel-tomo-model/src/tomography_python_programs/import_tilt_series/serialem.py", line 157, in _generate_tilt_image_dataframe df['date_time'] = pd.to_datetime(df['date_time'], infer_datetime_format=True) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in getitem indexer = self.columns.get_loc(key) File "/d/emr207/u/ubcg87b/miniforge/envs/relion/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc raise KeyError(key) from err KeyError: 'date_time'

alisterburt commented 1 year ago

hey! thanks for the report!

I saw that the CamelCase to snake_case conversion was a little weird for keys which had numbers decided that it was easier if they keys just always matched those in the file and no case conversion was done - will check the PR over at RELION but removing camel_to_snake=True and making the keys match those in the file (e.g. date_time -> DateTime) should be grand :)