Unused imports: The numpy library is imported but not used anywhere in the code. It can be safely removed.
Redundant function definitions: The extract function is called by both extract_sentbert and extract_originbert functions. Instead of defining two separate functions for these two cases, you can define a single function that takes the model as a parameter.
Redundant code: The dev parameter is used to control whether to store the embeddings for each caption separately or concatenate them and store for each key. However, this parameter is only used in the extract function, and not in extract_sentbert or extract_originbert. Therefore, the dev parameter can be removed from both extract_sentbert and extract_originbert functions.
Progress bar: The tqdm library is used to display a progress bar for the loop that iterates over the caption data frame. However, the progress bar is not shown if dev=False in the extract function. To show a progress bar in both cases, you can move the tqdm initialization to the beginning of the extract function, and use it in both cases.
Unused imports: The numpy library is imported but not used anywhere in the code. It can be safely removed.
Redundant function definitions: The extract function is called by both extract_sentbert and extract_originbert functions. Instead of defining two separate functions for these two cases, you can define a single function that takes the model as a parameter.
Redundant code: The dev parameter is used to control whether to store the embeddings for each caption separately or concatenate them and store for each key. However, this parameter is only used in the extract function, and not in extract_sentbert or extract_originbert. Therefore, the dev parameter can be removed from both extract_sentbert and extract_originbert functions.
Progress bar: The tqdm library is used to display a progress bar for the loop that iterates over the caption data frame. However, the progress bar is not shown if dev=False in the extract function. To show a progress bar in both cases, you can move the tqdm initialization to the beginning of the extract function, and use it in both cases.