search

roboflow / external-bugtracker

1 stars 2 forks source link

Annotator memory leak #15

Open alec-jensen opened 1 year ago

alec-jensen commented 1 year ago

Describe the bug After using the annotator with label assist for a while, the memory usage gets very large. The annotator becomes sluggish, eventually freezing altogether.

To Reproduce Steps to reproduce the behavior:

  1. Open the annotator
  2. Enable Label Assist
  3. Annotate for a while

Expected behavior The memory usage stays low, and the annotator does not slow down after time.

Evidence image Desktop (please complete the following information):

Edit: this looks like the same issue as #10 Edit 2: As said by narugo1992, windows does not crash. The memory leak still occurs on windows, but the annotator does not slow, freeze, or crash.

alec-jensen commented 1 year ago

I just had it crash in windows too.

lights0123 commented 1 year ago

Some Chromium logs when this happens:

[12106:21:0100/000000.533530:ERROR:cmd_buffer_helper.cc(141)] ContextResult::kFatalFailure: CommandBufferHelper::AllocateRingBuffer() failed
[12225:25:0100/000000.011712:ERROR:cmd_buffer_helper.cc(141)] ContextResult::kFatalFailure: CommandBufferHelper::AllocateRingBuffer() failed
[11906:11978:1019/131201.242229:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131201.242346:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131201.280123:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[11906:11978:1019/131201.280288:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131201.280334:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131201.280375:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[11906:11978:1019/131201.280442:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131201.280482:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131201.280519:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[11906:11906:1019/131201.282825:ERROR:shared_image_factory.cc(758)] DestroySharedImage: Could not find shared image mailbox
[11906:11906:1019/131201.282918:ERROR:shared_image_stub.cc(480)] SharedImageStub: Unable to destroy shared image
[11906:11906:1019/131201.331849:ERROR:shared_image_manager.cc(220)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[11906:11978:1019/131627.883831:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131627.883984:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131627.884055:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[11906:11978:1019/131627.884126:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131627.884189:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131627.884247:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[11906:11978:1019/131627.884358:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131627.884424:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131627.884483:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[11906:11978:1019/131627.884551:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131627.884611:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131627.884669:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[14511:25:0100/000000.637001:ERROR:cmd_buffer_helper.cc(141)] ContextResult::kFatalFailure: CommandBufferHelper::AllocateRingBuffer() failed
[11906:11978:1019/131821.156011:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131821.156100:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131821.156150:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[11906:11978:1019/131821.157679:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131821.157777:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131821.157834:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[11906:11978:1019/131821.157947:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131821.158025:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131821.158086:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[11906:11978:1019/131930.692033:ERROR:validation_errors.cc(106)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[11906:11978:1019/131930.692145:ERROR:gpu_child_thread.cc(67)] Mojo error in GPU process: Received bad user message: Validation failed for gpu.mojom.GpuChannel [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)]
[11906:11978:1019/131930.692231:ERROR:interface_endpoint_client.cc(702)] Message 2062245334 rejected by interface gpu.mojom.GpuChannel
[16335:25:0100/000000.326718:ERROR:cmd_buffer_helper.cc(141)] ContextResult::kFatalFailure: CommandBufferHelper::AllocateRingBuffer() failed
[11906:11906:1019/132348.960375:ERROR:shared_image_manager.cc(220)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[11906:11906:1019/132348.982594:ERROR:shared_image_manager.cc(220)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[11906:11906:1019/132348.988399:ERROR:shared_image_representation.cc(475)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (2560, 352)
[11906:11906:1019/132348.994057:ERROR:shared_image_representation.cc(475)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (2560, 352)
[11906:11906:1019/132348.999202:ERROR:shared_image_representation.cc(475)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (2560, 352)
[11906:11906:1019/132349.004089:ERROR:shared_image_manager.cc(220)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[11906:11906:1019/132349.004650:ERROR:shared_image_manager.cc(220)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[11906:11906:1019/132349.004827:ERROR:shared_image_representation.cc(475)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (320, 320)
[11906:11906:1019/132349.005402:ERROR:shared_image_representation.cc(475)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (64, 64)
[11906:11906:1019/132349.005489:ERROR:shared_image_representation.cc(475)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (320, 1280)
[11906:11906:1019/132349.007892:ERROR:shared_image_representation.cc(475)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (128, 64)
[11906:11906:1019/132349.008057:ERROR:shared_image_representation.cc(475)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (256, 64)
[11906:11906:1019/132349.008190:ERROR:shared_image_manager.cc(220)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[11906:11906:1019/132349.008999:ERROR:shared_image_representation.cc(475)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (384, 64)
[11906:11906:1019/132349.009243:ERROR:shared_image_manager.cc(220)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[1019/132349.053451:ERROR:process_memory_linux.cc(49)] pread64: Input/output error (5)

Interesting that this happens after just a few hundred MB of JS and GPU memory when I have many times that available.

josephofiowa commented 5 months ago

Hey, @alec-jensen, thanks for the report. (Cross posting my forum reply here in case others come across this issue) --

Hey, Alec, we’ve been aware of memory leaks — note: some isolated to Linux browsers — and delivered updates for many of them.

Here’s a summary of issues we isolated and pushed fixes for recently:

  1. The user’s browser doesn’t properly garbage collect
  2. WebGL contexts weren’t getting released
  3. SAM embeddings were being held
  4. Ctrl+Z history wasn’t bounded

Our team have been doing extensive memory leak debugging and fixes these last few weeks as well and has been able to verify with the profiler that memory usage no longer grows as you page through and annotate hundreds of images. This includes being able to scroll ~2000 images and not seeing any issues.

There is one known outstanding issue caused by react devtools. Disabling the chrome extension avoids this leak, should that affect you.

Can you confirm if your original message issue still persists for you?