MaartenGr
commented
1 month ago This mismatch between the actual implementation steps and the documented process can create confusion, especially for non-CS researchers (like myself) or contributors who rely on logging to understand the modularity and steps of the implementation.
Agreed. If things are unclear because I unnecessarily tried to simplify things, then we need to be explicit instead and more inline with the documentation.
Proposed Solution: To better reflect the detailed process, I propose to align it with the current documentation. A more detailed logging flow might look like:
Sounds good! I suggest we keep all logging before the representation steps and improve the logging at the representation level. I like separating the c-TF-IDF from additional representations (although c-TF-IDF can also be an additional representation/aspect).
Conceptual Clarification: It also comes to play that some concepts are still ambiguous an can be approached differently. For example, the definition of “topic” can be seen in two ways:
As the clustering output, where documents are grouped and assigned a topic ID (but there are no topic keywords defined). At the end of the c-TF-IDF step, where top-n-keywords provide a more concrete representation or “definition” of each topic.
I generally try to refer to a "topic" as the output of any representation, whether that is c-TF-IDF. Implicitly, that does contain the output of the clustering but is not sufficient to me. So technically, I would say at the end of the c-TF-IDF where I attempt to extract topics from clusters.
As such, the statement of "extracting topics from" is not a precise term, so I would avoid it in this new scheme, unless there is a clear cut point where topics are "defined".
But yeah, this definitely makes sense here since the definition of a "topic" has also shifted a bit with the introduction of additional representations, multimodal topic modeling, etc.
All in all, sounds good to me! Thank you for considering this and taking the time 😄
PipaFlores
Feature request
Following some reflection based on a closed PR, I want to suggest a change in the logger.info messages to align with the documentation and modular pipeline of the library.
Context: In the current implementation, the code's logging verbosity reflects a simplified version of the pipeline:
Embeddings -> Dim. Red. -> Clustering -> Representation
However, in the documentation and associated research papers, the pipeline is presented with more detailed later stages:
Embeddings -> Dim. Red. -> Clustering -> Bag of Words (vectorizer + c-TF-IDF) -> Fine Tuning Representations
This mismatch between the actual implementation steps and the documented process can create confusion, especially for non-CS researchers (like myself) or contributors who rely on logging to understand the modularity and steps of the implementation.
Problem Description:
There’s ambiguity in the logging related to the term “Representation,” which might be interpreted differently depending on the pipeline step.
The pipeline section involving the c-TF-IDF is currently not logged at all, despite being a core part of the pipeline.
The logging output skips from:
Proposed Solution: To better reflect the detailed process, I propose to align it with the current documentation. A more detailed logging flow might look like:
Conceptual Clarification: It also comes to play that some concepts are still ambiguous an can be approached differently. For example, the definition of “topic” can be seen in two ways:
As such, the statement of "extracting topics from" is not a precise term, so I would avoid it in this new scheme, unless there is a clear cut point where topics are "defined".
Given that the c-TF-IDF step plays a foundational role in refining topic representations, I suggest treating it as a separate and essential component in the pipeline and reflecting this in the logger output.
Motivation
I want to improve this library, since I will probably be using during my research (already used it for a paper).
Your contribution
I can implement the solution, if agreed. I will probably start doing so for my own purposes, so a commit id will be available here for observation/comments when done.