[Bug]: loading 27k images for lora training fails to load all to dataset tools

[modpotato]

What happened?

attempted to open dataset tools to automatically tag all images loaded, tkinter fails to create enough menus for them on the left where you can change between them.

What did you expect would happen?

menus would load properly and handle tagging at scale

Relevant log output

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\md\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 1948, in __call__
    return self.func(*args)
           ^^^^^^^^^^^^^^^^
  File "D:\OneTrainer\venv\Lib\site-packages\customtkinter\windows\widgets\ctk_button.py", line 554, in _clicked
    self._command()
  File "D:\OneTrainer\modules\ui\CaptionUI.py", line 500, in open_directory
    self.load_directory(include_subdirectories=self.config_ui_data["include_subdirectories"])
  File "D:\OneTrainer\modules\ui\CaptionUI.py", line 216, in load_directory
    self.file_list_column(self.bottom_frame)
  File "D:\OneTrainer\modules\ui\CaptionUI.py", line 152, in file_list_column
    label.grid(row=i, column=0, padx=5, sticky="nsw")
  File "D:\OneTrainer\venv\Lib\site-packages\customtkinter\windows\widgets\core_widget_classes\ctk_base_class.py", line 321, in grid
    return super().grid(**self._apply_argument_scaling(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\md\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 2549, in grid_configure
    self.tk.call(
_tkinter.TclError: row out of bounds
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\md\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 1948, in __call__
    return self.func(*args)
           ^^^^^^^^^^^^^^^^
  File "D:\OneTrainer\venv\Lib\site-packages\customtkinter\windows\widgets\ctk_button.py", line 554, in _clicked
    self._command()
  File "D:\OneTrainer\modules\ui\CaptionUI.py", line 508, in open_caption_window
    dialog = GenerateCaptionsWindow(self, self.dir, self.config_ui_data["include_subdirectories"])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\OneTrainer\modules\ui\GenerateCaptionsWindow.py", line 57, in __init__
    self.mode_dropdown = ctk.CTkOptionMenu(self.frame, variable=self.mode_var, values=self.modes, dynamic_resizing=False, width=200)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\OneTrainer\venv\Lib\site-packages\customtkinter\windows\widgets\ctk_optionmenu.py", line 86, in __init__
    self._dropdown_menu = DropdownMenu(master=self,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\OneTrainer\venv\Lib\site-packages\customtkinter\windows\widgets\core_widget_classes\dropdown_menu.py", line 25, in __init__
    tkinter.Menu.__init__(self, *args, **kwargs)
  File "C:\Users\md\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 3333, in __init__
    Widget.__init__(self, master, 'menu', cnf, kw)
  File "C:\Users\md\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 2628, in __init__
    self.tk.call(
_tkinter.TclError: No more menus can be allocated.
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\md\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 1948, in __call__
    return self.func(*args)
           ^^^^^^^^^^^^^^^^
  File "D:\OneTrainer\venv\Lib\site-packages\customtkinter\windows\ctk_toplevel.py", line 91, in destroy
    tkinter.Toplevel.destroy(self)
  File "C:\Users\md\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 2635, in destroy
    for c in list(self.children.values()): c.destroy()
                                           ^^^^^^^^^^^
  File "D:\OneTrainer\venv\Lib\site-packages\customtkinter\windows\widgets\core_widget_classes\ctk_base_class.py", line 98, in destroy
    tkinter.Frame.destroy(self)
  File "C:\Users\md\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 2635, in destroy
    for c in list(self.children.values()): c.destroy()
                                           ^^^^^^^^^^^
  File "D:\OneTrainer\venv\Lib\site-packages\customtkinter\windows\widgets\ctk_optionmenu.py", line 178, in destroy
    self._variable.trace_remove("write", self._variable_callback_name)
  File "C:\Users\md\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 471, in trace_remove
    self._tk.call('trace', 'remove', 'variable',
_tkinter.TclError: wrong # args: should be "trace remove variable name opList command"

Output of pip freeze

distlib==0.3.7 filelock==3.13.1 platformdirs==4.1.0 pypresence==4.3.0 virtualenv==20.25.0

[311-code]

I also have a side issue sort of related. When trying to train 113,246 images, if one single image is bad it messes up the latent caching, and have to delete it and start over. This isn't ideal and should really find a way to bypass corrupted images automatically.

could not load image, it might be corrupted: H:/Dataset\145439\2966342.jpg | 20652/113246 [44:55<3:02:18, 8.47it/s] caching: 18%|███████████▋ | 20652/113246 [44:55<3:21:23, 7.66it/s] epoch: 0%| | 0/180 [45:03<?, ?it/s] Traceback (most recent call last): File "C:\OneTrainer\modules\ui\TrainUI.py", line 478, in __training_thread_function trainer.train()