bghira
commented
1 day ago your system might be haunted. the respective code makes more sense than the error. let me show:
def batch_write_embeddings(self):
"""Process write requests in batches."""
batch = []we're initialising batch here to an empty list.
written_elements = 0
while True:
try:
# Block until an item is available or timeout occurs
first_item = self.write_queue.get(timeout=1)
batch = [first_item]here it is additionally not deleted but is set to a populated list of the first item.
# Try to get more items without blocking
while (
not self.write_queue.empty() and len(batch) < self.write_batch_size
):this is the line where your error occurs ^
but batch is an empty list or populated with one item.
maybe later in the loop it del the batch:
logger.debug("Retrieving more items from the queue.")
items = self.write_queue.get_nowait()
batch.append(items)
logger.debug(f"Batch now contains {len(batch)} items.")
self.process_write_batch(batch)
self.write_thread_bar.update(len(batch))
logger.debug("Processed batch write.")
written_elements += len(batch)
except queue.Empty:
# Timeout occurred, no items were ready
if not self.process_write_batches:
if len(batch) > 0:
self.process_write_batch(batch)
self.write_thread_bar.update(len(batch))
logger.debug(
f"Exiting batch write thread, no more work to do after writing {written_elements} elements"
)
break
logger.debug(
f"Queue is empty. Retrieving new entries. Should retrieve? {self.process_write_batches}"
)
pass
except Exception:
logger.exception("An error occurred while writing embeddings to disk.")
logger.debug("Exiting background batch write thread.")nope. no deletions of the batch variable.
it might be a bad RunPod instance?
playerzer0x
I'm training Flux on multi-gpu Runpod instances and receiving this UnboundLocalError during training start-up, and upon checkpoints saves. Training continues thankfully, but sharing in case it needs to be fixed: