Codecarbon keep running even when function fails

[abhishekkrthakur]

• CodeCarbon version: 2.2.0
• Python version: 3.8.x
• Operating System: Ubuntu 22.04

As the title says. The function crashes but codecarbon keeps running. this is quite undesirable if codecarbon has to be used in a production environment. For example, a training ran for 3 days just because code carbon didnt shut down after function crash. Is it possible to fix it soon?

[benoit-cty]

Hello, Can you share the way you call it ? If you use the decorator it should work. If you use tracker.start(), you have to do a tracker.stop() in the try/except.

[abhishekkrthakur]

yeah. not using decorator.

[benoit-cty]

The problem is that we use a scheduler and I don't know how to let it know that the main program has crashed without using the decorator. We need to had this in the documentation.

try:
    # Compute intensive code goes here
    a = 1 + 1
finally:
    emissions = tracker.stop()
print(f"{emissions} kgCO2eq")