search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
613 stars 222 forks source link

Can you please fix log file and output level once and for all ! #2820

Open flabellle-fn opened 1 month ago

flabellle-fn commented 1 month ago

Which version of the AzCopy was used?

10.26.0

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

AzCopy copy ... --log-level XXXX --output-level XXX (a perfectly working command, and then, the same with invalid SAS to generate an error and test various log level combinations).

What problem was encountered?

  1. The log files are annoying, useless and especially does not respect log levels.
  2. --log-level ERROR does not work, it logs anyway.... error or not.
  3. when --log-level NONE is use, no log files are generated. Congrats ! But, on the output, you are falsely reporting 'the log file is...' !!!
  4. When output-level is set to QUIET and an error occurs, it displays anyway, but completely useless information for end user, like:

panic: close of nil channel

goroutine 89 [running]: github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).reportJobPartDoneHandler(0xc000516008) /mnt/vss/_work/1/s/ste/mgr-JobMgr.go:718 +0x327 created by github.com/Azure/azure-storage-azcopy/v10/ste.NewJobMgr in goroutine 1 /mnt/vss/_work/1/s/ste/mgr-JobMgr.go:203 +0xc88

  1. There is no output-level ERROR switch that would be useful. (What is ESSENTIAL anyway ?!?!?)
  2. xxx-scanning.log file are useless and cannot be turned off.
  3. There is no way to log into ONLY one file... (other than setting the log-level to NONE and capturing the output.
  4. There is no timestamp, that's alway a must in log files (or output that you redirect in a file).

You want to know when everything will be fine ? It's when you will be able to provide a way to log ONLY errors, the significative part of it, for an end user, to a single file. E.g. command line and the error encountered.

How can we reproduce the problem in the simplest way?

Just try it...

Have you found a mitigation/solution?

NO !

Thanks

dphulkar-msft commented 4 weeks ago

Hi @flabellle-fn

Thank you for your feedback. We'd like to clarify a few points regarding the logging behavior:

  1. Timestamps in Logs: The log file correctly includes timestamps for every log entry, and this feature is working as expected.

  2. Log Levels: When --log-level is set to ERROR, only error logs and critical job-related information are logged, as intended.

  3. Console Output with --output-level=QUIET: In this case, no output should appear in the console. If you're seeing any output, it might be due to a panic being generated, which is meant to assist with debugging but should not be visible to end users. We'll review this to ensure proper handling.

  4. When --log-level is set to NONE, the xxx-scanning.log and the job log file is not created, by doing so you can avoid creating the log files.

Additionally, we're planning to address the following in an upcoming update:

  1. If --log-level is set to NONE, the log file location is currently printed on the console. We plan to address this so it won't be shown.

We appreciate your valuable insights and are working to enhance the experience based on your feedback. Please feel free to reach out if you have any other concerns.