paul-tqh-nguyen / arxiv_as_a_newspaper

arxiv.org portrayed as if it were a news paper.
0 stars 0 forks source link

Type conflict in function signature of _recent_page_url_info_tuple_to_dict #10

Closed paul-tqh-nguyen closed 5 years ago

paul-tqh-nguyen commented 5 years ago

Here's the stack:

Traceback (most recent call last):
  File "./arxiv_as_a_newspaper.py", line 116, in <module>
    main()
  File "./arxiv_as_a_newspaper.py", line 103, in main
    run_etl_process()
  File "./arxiv_as_a_newspaper.py", line 34, in run_etl_process
    etl_processing_utilities.run_etl_process()
  File "/home/pnguyen/code/arxiv_as_a_newspaper/utilities/etl_processing_utilities.py", line 40, in run_etl_process
    number_of_documents_stored = load_utilities.write_documents_as_dicts_to_arxiv_recent_papers_collection(dicts_to_store)
  File "/home/pnguyen/code/arxiv_as_a_newspaper/utilities/load_utilities.py", line 138, in write_documents_as_dicts_to_arxiv_recent_papers_collection
    dicts = list(dicts)
  File "/home/pnguyen/code/arxiv_as_a_newspaper/utilities/extract_transform_utilities.py", line 302, in _expand_dicts_to_store_with_research_field
    for dict_to_store in dicts_to_store:
  File "/home/pnguyen/code/arxiv_as_a_newspaper/utilities/extract_transform_utilities.py", line 192, in _recent_page_url_info_tuple_to_dict
    author_to_author_link_dictionaries = _dwim_author_to_author_link_dictionary_wrt_mirror_link(author_to_author_link_dictionaries)
NameError: name '_dwim_author_to_author_link_dictionary_wrt_mirror_link' is not defined
paul-tqh-nguyen commented 5 years ago

Patch: https://github.com/paul-tqh-nguyen/arxiv_as_a_newspaper/commit/3afff2f50770549d2b60aae634834d4c82cb16dd

This patch will fix it.

We pass around a list of dicts of the form { AUTHOR : AUTHOR_LINK }.

When transforming this list of dicts into JSON as part of our ETL, we used to have a type conflict where we'd expect single dicts in _dwim_author_to_author_link_dictionary_wrt_mirror_link.

This led to our ETL not working.

This patch refactors _dwim_author_to_author_link_dictionary_wrt_mirror_link into _dwim_author_to_author_link_dictionaries_wrt_mirror_link (it does what it says it does, which is fix this issue entirely).