why AzcopyVersion 10.18.1 doesnot show detailed log even after defining log verbosity with --log-level INFO for the log file

damison1 commented 6 days ago

Which version of the AzCopy was used? AzcopyVersion 10.18.1

Note: The version is visible when running AzCopy without any argument

Which platform are you using? (ex: Windows, Mac, Linux) LINUX

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

cat "/path/to/file.txt" | azcopy cp "https://[account].blob.core.windows.net/[container]/[path/to/blob]?[SAS]" --from-to PipeBlob --log-level

What problem was encountered?

No detailed log $ cat 4fee856d-db1b-3347-6aec-14eb756b7486.log 2024/09/13 09:54:40 AzcopyVersion 10.18.1 2024/09/13 09:54:40 OS-Environment linux 2024/09/13 09:54:40 OS-Architecture amd64 2024/09/13 09:54:40 Log times are in UTC. Local time is 13 Sep 2024 09:54:40 2024/09/13 09:54:40 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2024-09-13T09:54:35Z or --include-after=2024-09-13T09:54:35Z

How can we reproduce the problem in the simplest way?

Have you found a mitigation/solution?

When I run higher version 10.25.1, I can see detailed log like PUT operation and all but we don't want to upgrade to higher version and need detailed log with current azcopy version that we are using i.e. 10.18.1

damison1 commented 6 days ago

cat "/path/to/file.txt" | azcopy cp "https://[account].blob.core.windows.net/[container]/[path/to/blob]?[SAS]" --from-to PipeBlob --log-level INFO

vibhansa-msft commented 3 days ago

10.18 is quite an older version. Kindly upgrade to the latest version and see if logging is working fine or not.

damison1 commented 3 days ago

Hi @vibhansa-msft,

we don't want to upgrade azcopy version and keeping the same version is there any possible way that we can get detailed logs.. Let me know if we can define any environment variable or anything like this to get detailed azcopy log for 10.18.1 azcopy version

Command I run: cat /path/to/file | azcopy cp "https://storageaccount.blob.core.windows.net/container/datavolume_0000.dat?SAS" --from-to PipeBlob --log-level INFO

Difference in logs xbrhdph10101:HDB:ph1adm ~/.azcopy 68> cat 6cdcc629-3218-4547-4cca-8444ee306525.log 2024/09/13 09:38:45 AzcopyVersion 10.18.1 2024/09/13 09:38:45 OS-Environment linux 2024/09/13 09:38:45 OS-Architecture amd64 2024/09/13 09:38:45 Log times are in UTC. Local time is 13 Sep 2024 09:38:45 2024/09/13 09:38:45 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2024-09-13T09:38:40Z or --include-after=2024-09-13T09:38:40Z

xbrhdph10101:HDB:ph1adm ~/.azcopy 63> cat 01a989f9-23f8-3c4b-5149-d9d25f35a5c2.log 2024/09/13 11:33:23 AzcopyVersion 10.25.1 2024/09/13 11:33:23 OS-Environment linux 2024/09/13 11:33:23 OS-Architecture amd64 2024/09/13 11:33:23 Log times are in UTC. Local time is 13 Sep 2024 11:33:23 2024/09/13 11:33:23 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2024-09-13T11:33:18Z or --include-after=2024-09-13T11:33:18Z 2024/09/13 11:33:24 ==> REQUEST/RESPONSE (Try=1/44.8µs, OpTime=63.203037ms) -- RESPONSE STATUS CODE ERROR PUT https://xbr0201backuptsi0001.blob.core.windows.net/xbr0201/datavolume_0000.dat?blockid=Me734ivkQFNBey2TBdCfPQAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%3D%3D&comp=block&se=2024-09-14T18%3A18%3A21Z&sig=-REDACTED-&sp=rwdlacup&spr=https&srt=sco&ss=b&sv=2021-06-08 Accept: application/xml Content-Length: 8388608 Content-Type: application/octet-stream User-Agent: AzCopy/10.25.1 azsdk-go-azblob/v1.3.1 (go1.22.4; linux) X-Ms-Client-Request-Id: 0c57a556-1252-4d6d-6f96-4c274bd3c354 x-ms-version: 2023-08-03 RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. Content-Length: 408 Content-Type: application/xml Date: Fri, 13 Sep 2024 11:33:23 GMT Server: Microsoft-HTTPAPI/2.0 X-Ms-Error-Code: AuthenticationFailed X-Ms-Request-Id: 05939181-801e-0096-62d0-0588b6000000 Response Details: AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. Signature fields not well formed.</

vibhansa-msft commented 1 day ago

Try using log-level to debug instead of info and you may get more detailed logs.

damison1 commented 15 hours ago

Hi @vibhansa-msft,

Tried but it's also not working

cat "/path/to/file.txt" | azcopy cp "https://[account].blob.core.windows.net/[container]/[path/to/blob]?[SAS]" --from-to PipeBlob --log-level DEBUG

azthdbh20201:HDB:bh2adm ~/.azcopy 55> cat 457a6db6-768d-e643-47fa-279bd84afef3.log 2024/09/16 13:01:52 AzcopyVersion 10.18.1 2024/09/16 13:01:52 OS-Environment linux 2024/09/16 13:01:52 OS-Architecture amd64 2024/09/16 13:01:52 Log times are in UTC. Local time is 16 Sep 2024 13:01:52 2024/09/16 13:01:52 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2024-09-16T13:01:47Z or --include-after=2024-09-16T13:01:47Z

Azure / azure-storage-azcopy