search

brainglobe / cellfinder

Automated 3D cell detection in very large images
https://brainglobe.info/documentation/cellfinder/index.html
BSD 3-Clause "New" or "Revised" License
181 stars 39 forks source link

[BUG] Training stalls at the end of the first epoch with keras 3.6.0 #463

Open IgorTatarnikov opened 1 month ago

IgorTatarnikov commented 1 month ago

Training seems to enter an infinite loop, or wait for a callback that's not signalled at the end of an epoch. This causes training to stall at the end of the first epoch.

See https://github.com/brainglobe/cellfinder/actions/runs/11209740840

Downgrading keras to 3.5.0 fixes this issue. #462 pinned this version temporarily.

alessandrofelder commented 1 month ago

@IgorTatarnikov has volunteered to investigate - thank you!

IgorTatarnikov commented 2 weeks ago

https://github.com/keras-team/keras/issues/20425 Should be fixed in keras main