Quiet all output for load_and_run_detector_batch

[ThomasLuypaert]

Hi all,

I am running the MegaDetector model on many folders containing many images, using a Jupyter Notebook in JupyterLabs on my university's HPC.

The part of the script that runs the detection part looks as follows:

import os
import sys
import io
from tqdm import tqdm
from contextlib import redirect_stdout, redirect_stderr
from megadetector.utils.path_utils import write_list_to_file
from megadetector.detection.run_detector_batch import load_and_run_detector_batch, write_results_to_file

# Grab folders that need processing
folders_for_processing = [[os.path.join(folder, file) for file in os.listdir(folder)] for folder in frame_folders]

# Create output folders to store the data
folders_for_output = [[file.replace("Frames", "Detection_output") for file in folder] for folder in folders_for_processing]
[[os.makedirs(file, exist_ok=True) for file in folders] for folders in folders_for_output]

# Create empty '.json' files to store the data
file_basenames = [[os.path.basename(file).replace(".mp4", "") for file in folder] for folder in folders_for_output]
output_files = [[os.path.join(folders_for_output[i][j],'{}.json'.format(file_basenames[i][j])) for j in range(len(folders_for_output[i]))] for i in range(len(folders_for_output))]

[[write_list_to_file(output_file = file, strings = "") for file in folder] for folder in output_files]

# Run MegaDetector on each folder containing frames

out = io.StringIO()
err = io.StringIO()

for i in range(len(folders_for_processing)):
        for j in range(len(folders_for_processing[i])):

            with redirect_stdout(out), redirect_stderr(err):
                json = load_and_run_detector_batch(model_file="MDV5A",
                                                   image_file_names=folders_for_processing[i][j], 
                                                   quiet = True)
            with redirect_stdout(out), redirect_stderr(err):
                 write_results_to_file(results=json, 
                                        output_file=output_files[i][j],
                                       detector_file="MDV5A")

The script seems to be working as expected, however, despite my best efforts to quiet the output produced by these functions, the following output is still being produced:

Is there a way to turn off all the output, so I can implement my own progress bar that doesn't produce so much output (e.g. only update the bar for each folder being processed, not for each file).

Apologies if I am missing something, I tried finding advice online but I cannot seem to capture all the output.

Best regards, Thomas

[agentmorris]

My first suggestion would be to ignore these remaining printouts if they're not a huge inconvenience; they will only happen during initialization, so they shouldn't interfere with your progress reporting. How important is it to eliminate these?

But, FWIW, these printouts come from within the YOLOv5 code that MegaDetector uses as a supporting layer, e.g. here.

That means that if you are running MegaDetector via the MegaDetector Python package, the way we demonstrate in our sample notebook, it will be somewhat difficult to suppress these printouts, since they happen in an imported library.

If you are running MegaDetector by cloning the MegaDetector and YOLOv5 repos, you can search the YOLOv5 repo for "fusing layers" and "model summary", and comment out the corresponding lines, which I believe are just here and here.

But, I'm doubling down on what I said above, suppressing these last couple printouts (which shouldn't get in the way of progress reporting) may not be worth depending on that kind of one-off customization, it may make installation a hassle later.

Let me know if that helps!

[ThomasLuypaert]

Hi!

Yes, that helps, thank you!

It is not incredibly important to suppress these, it is more of an inconvenience than a real issue, since the output crowds my workspace (it seems the initialization output occurs every time a new folder is processed, and I have many folders). Thanks for providing some insights into where these outputs come from an why my attempts to suppress them have failed.

Cheers, Thomas