search

mlco2 / codecarbon

Track emissions from Compute and recommend ways to reduce their impact on the environment.
https://mlco2.github.io/codecarbon
MIT License
1.02k stars 158 forks source link

No output from EmissionsTracker #370

Closed coillarach closed 1 year ago

coillarach commented 1 year ago

Description

I Installed CodeCarbon according to the instructions as part of a JupyterHub setup. The example code worked perfectly with several groups of students, but then the output from the package just stopped. The data was no longer written to the emissions.csv file, and there was no output to the console on calling tracker.stop(). I have exactly the same configuration running locally where it still works fine. I therefore suspect that there is something about the state of the server installation that is causing the problem. My question is, under what circumstances would this behaviour be expected? That would help me track down the problem. From the CodeCarbon point of view, the issue is that there is no error message to say what has failed.

What I Did

Installed TLJH: https://tljh.jupyter.org/en/latest/install/index.html Installed required packages including CodeCarbon Prepared and tested the notebook which worked perfectly using the examples: https://mlco2.github.io/codecarbon/examples.html Ran the same session for several groups of roughly 20 students - everything was fine for the first three groups, but on the fourth, EmissionsTracker.stop() returned no output and nothing was written to emissions.csv

I have reduced the number of configured users and reduced the volume of files to free up disk space. Neither of these actions worked. I have also restarted the TLJH server and reinstalled CodeCarbon but the behaviour is the same.

from codecarbon import EmissionsTracker
import timeit
tracker = EmissionsTracker()
tracker.start()
t1 = timeit.timeit(stmt='[random.randint(0, 1000) for r in range(10000)]', 
              setup='import random', 
              number=1000
             )
tracker.stop()
benoit-cty commented 1 year ago

Thanks for reporting this. Did you see the log anywhere ?

FYI using CodeCarbon multiple time on the same server at the same time will report wrong result as it will compute the server footprint for each notebook, not only the notebook running it. We are working on https://github.com/mlco2/codecarbon/pull/316 for that.

coillarach commented 1 year ago

Hi Benoit

Thanks for getting back to me so quickly.

I haven’t seen a logfile other than the emissions.csv – can you suggest where to look or what the name of the logfile might be? I have checked the notebook working directory but there is nothing there.

Brian

Dr Brian Davison MA MSc PhD SFHEA Lecturer & Teaching Fellow School of Computing Edinburgh Napier University 10 Colinton Road Edinburgh EH10 5DT Scotland, UK

+44 (0)131 4552373

From: Benoit Courty @.> Date: Tuesday, 7 February 2023 at 21:23 To: mlco2/codecarbon @.> Cc: Davison, Brian @.>, Author @.> Subject: Re: [mlco2/codecarbon] No output from EmissionsTracker (Issue #370) ​

CAUTION: This email originated from outside Edinburgh Napier University. Do not follow links or open attachments if you doubt the authenticity of the sender or the content.

Thanks for reporting this. Did you see the log anywhere ?

FYI using CodeCarbon multiple time on the same server at the same time will report wrong result as it will compute the server footprint, not only the notebook. We are working on #316https://github.com/mlco2/codecarbon/pull/316 for that.

— Reply to this email directly, view it on GitHubhttps://github.com/mlco2/codecarbon/issues/370#issuecomment-1421470803, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACZZDMSPRJLRJMT6WSUDQ7LWWK4OPANCNFSM6AAAAAAUO5ZCA4. You are receiving this because you authored the thread.Message ID: @.***>

This message and its attachment(s) are intended for the addressee(s) only and should not be read, copied, disclosed, forwarded or relied upon by any person other than the intended addressee(s) without the permission of the sender. If you are not the intended addressee you must not take any action based on this message and its attachment(s) nor must you copy or show them to anyone. Please respond to the sender and ensure that this message and its attachment(s) are deleted.

It is your responsibility to ensure that this message and its attachment(s) are scanned for viruses or other defects. Edinburgh Napier University does not accept liability for any loss or damage which may result from this message or its attachment(s), or for errors or omissions arising after it was sent. Email is not a secure medium. Emails entering Edinburgh Napier University's system are subject to routine monitoring and filtering by Edinburgh Napier University.

Edinburgh Napier University is a registered Scottish charity. Registration number SC018373

BSL users can contact us via contactSCOTLAND-BSL, the on-line British Sign Language interpreting service. Find out more on the contactSCOTLAND website.

benoit-cty commented 1 year ago

Hi Brian, I just do a test on a new project in TLJH and I saw the log in the Notebook:

image

jupyter-lab --version 3.5.3 jupyterhub --version 1.4.0

coillarach commented 1 year ago

Hi Benoit

I don’t get that much log output:

[Graphical user interface, text, application Description automatically generated]

Is there a ‘verbose’ setting that I can enable?

Brian

Dr Brian Davison MA MSc PhD SFHEA Lecturer & Teaching Fellow School of Computing Edinburgh Napier University 10 Colinton Road Edinburgh EH10 5DT Scotland, UK

+44 (0)131 4552373

From: Benoit Courty @.> Date: Friday, 10 February 2023 at 20:46 To: mlco2/codecarbon @.> Cc: Davison, Brian @.>, Author @.> Subject: Re: [mlco2/codecarbon] No output from EmissionsTracker (Issue #370) ​

CAUTION: This email originated from outside Edinburgh Napier University. Do not follow links or open attachments if you doubt the authenticity of the sender or the content.

Hi Brian, I just do a test on a new project in TLJH and I saw the log in the Notebook:

[image]https://user-images.githubusercontent.com/6603048/218193915-50ab3928-1a37-425a-a707-8ff9398d3df3.png

jupyter-lab --version 3.5.3 jupyterhub --version 1.4.0

— Reply to this email directly, view it on GitHubhttps://github.com/mlco2/codecarbon/issues/370#issuecomment-1426326647, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACZZDMSYWXZZQCJIH7IU3IDWW2SLZANCNFSM6AAAAAAUO5ZCA4. You are receiving this because you authored the thread.Message ID: @.***>

This message and its attachment(s) are intended for the addressee(s) only and should not be read, copied, disclosed, forwarded or relied upon by any person other than the intended addressee(s) without the permission of the sender. If you are not the intended addressee you must not take any action based on this message and its attachment(s) nor must you copy or show them to anyone. Please respond to the sender and ensure that this message and its attachment(s) are deleted.

It is your responsibility to ensure that this message and its attachment(s) are scanned for viruses or other defects. Edinburgh Napier University does not accept liability for any loss or damage which may result from this message or its attachment(s), or for errors or omissions arising after it was sent. Email is not a secure medium. Emails entering Edinburgh Napier University's system are subject to routine monitoring and filtering by Edinburgh Napier University.

Edinburgh Napier University is a registered Scottish charity. Registration number SC018373

BSL users can contact us via contactSCOTLAND-BSL, the on-line British Sign Language interpreting service. Find out more on the contactSCOTLAND website.

benoit-cty commented 1 year ago

You could add file logging this way:

tracker = EmissionsTracker()

# logger customization must be done after EmissionsTracker() because it reset all the log handlers.
set_logger_level(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler("codecarbon.log")
fh.setLevel(logging.DEBUG)
formatter = logging.Formatter(
    "%(asctime)s - %(name)-12s: %(levelname)-8s %(message)s"
)
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.debug("DEBUG TEST")
logger.info("INFO TEST")
logger.warning("warning TEST")

tracker.start()

You will get a file codecarbon.log in the notebook folder. It won't capture the initialization, that's our fault : one more issue.

coillarach commented 1 year ago

Hi Benoit

Thanks for the logging code.

However, when I tried it, everything worked as normal – the problem had vanished!

As far as I know, nothing has changed since the last time I tried and there was no output. I don’t think it’s the logging code that has resolved the issue because I tried restarting the kernel and running the original code again without logging, and again it worked perfectly.

I guess I will just have to wait until the problem re-occurs and then try looking at the debug output. I’ll keep you updated.

Thanks for your help Brian

Dr Brian Davison MA MSc PhD SFHEA Lecturer & Teaching Fellow School of Computing Edinburgh Napier University 10 Colinton Road Edinburgh EH10 5DT Scotland, UK

+44 (0)131 4552373

From: Benoit Courty @.> Date: Saturday, 11 February 2023 at 21:39 To: mlco2/codecarbon @.> Cc: Davison, Brian @.>, Author @.> Subject: Re: [mlco2/codecarbon] No output from EmissionsTracker (Issue #370) ​

CAUTION: This email originated from outside Edinburgh Napier University. Do not follow links or open attachments if you doubt the authenticity of the sender or the content.

You could add file logging this way:

tracker = EmissionsTracker()

logger customization must be done after EmissionsTracker() because it reset all the log handlers.

set_logger_level(logging.DEBUG)

create file handler which logs even debug messages

fh = logging.FileHandler("codecarbon.log")

fh.setLevel(logging.DEBUG)

formatter = logging.Formatter(

"%(asctime)s - %(name)-12s: %(levelname)-8s %(message)s"

)

fh.setFormatter(formatter)

logger.addHandler(fh)

logger.debug("DEBUG TEST")

logger.info("INFO TEST")

logger.warning("warning TEST")

tracker.start()

You will get a file codecarbon.log in the notebook folder. It won't capture the initialization, that's our fault : one more issue.

— Reply to this email directly, view it on GitHubhttps://github.com/mlco2/codecarbon/issues/370#issuecomment-1426881884, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACZZDMQABXIQK3YLJT2MNE3WXABJTANCNFSM6AAAAAAUO5ZCA4. You are receiving this because you authored the thread.Message ID: @.***>

This message and its attachment(s) are intended for the addressee(s) only and should not be read, copied, disclosed, forwarded or relied upon by any person other than the intended addressee(s) without the permission of the sender. If you are not the intended addressee you must not take any action based on this message and its attachment(s) nor must you copy or show them to anyone. Please respond to the sender and ensure that this message and its attachment(s) are deleted.

It is your responsibility to ensure that this message and its attachment(s) are scanned for viruses or other defects. Edinburgh Napier University does not accept liability for any loss or damage which may result from this message or its attachment(s), or for errors or omissions arising after it was sent. Email is not a secure medium. Emails entering Edinburgh Napier University's system are subject to routine monitoring and filtering by Edinburgh Napier University.

Edinburgh Napier University is a registered Scottish charity. Registration number SC018373

BSL users can contact us via contactSCOTLAND-BSL, the on-line British Sign Language interpreting service. Find out more on the contactSCOTLAND website.