Fix preprocessing with num_processes argument

MIND-Lab / OCTIS

OCTIS: Comparing Topic Models is Simple! A python package to optimize and evaluate topic models (accepted at EACL2021 demo track)

MIT License

705 stars 98 forks source link

Fix preprocessing with num_processes argument #96

Closed adaminsky closed 1 year ago

adaminsky commented 1 year ago

Previously, setting num_processes to anything other than None resulted in an error. This was due to the simple_preprocessing_steps function expecting a list of documents as an argument, but the process_map function applied it to every document in the corpus. This change standardizes the use of simple_preprocessing_steps to operate over a single document which allows it to be used with map and process_map. In addition, this adds a progress bar for when num_processes=None.

silviatti commented 1 year ago

Hi, so sorry for the late response. Thanks a lot for your contribution! This should solve issue #99. I'll merge it now. Thanks again :)

Silvia