search

taers232c / GAMADV-XTD3

Command line tool to manage Google Workspace
729 stars 85 forks source link

[BUG] csv_output_timestamp_column and formatjson don't play well together #262

Closed jay-eleven closed 2 years ago

jay-eleven commented 2 years ago

Hi Ross!!

I've noticed that csv_output_timestamp_column works as expected:

jay@cloudshell:~$ gam config csv_output_timestamp_column timestamp print domains
domainName,parentDomainName,creationTime,type,verified,timestamp
xxx,xxx,xxx,xxx,xxx,2022-04-16T20:11:00+00:00
yyy,yyy,yyy,yyy,yyy,2022-04-16T20:11:00+00:00

However, when combined with formatjson gam fails:

jay@cloudshell:~$ gam config csv_output_timestamp_column timestamp print domains formatjson
Traceback (most recent call last):
  File "__init__.py", line 61974, in ProcessGAMCommand
  File "__init__.py", line 13720, in doPrintShowDomains
  File "__init__.py", line 7584, in writeCSVfile
  File "__init__.py", line 7297, in writeCSVToStdout
  File "__init__.py", line 7275, in writeCSVData
  File "csv.py", line 157, in writerows
  File "csv.py", line 149, in _dict_to_list
ValueError: dict contains fields not in fieldnames: 'timestamp'

I've tried several gam commands randomly and all of them exhibit the same behaviour so I think this might be affecting all of them:

jay@cloudshell:~$ gam config csv_output_timestamp_column timestamp print users formatjson
Getting all Users, may take some time on a large Google Workspace Account...
Got x Users: a - b
Traceback (most recent call last):
  File "__init__.py", line 61974, in ProcessGAMCommand
  File "__init__.py", line 37479, in doPrintUsers
  File "__init__.py", line 7584, in writeCSVfile
  File "__init__.py", line 7297, in writeCSVToStdout
  File "__init__.py", line 7275, in writeCSVData
  File "csv.py", line 157, in writerows
  File "csv.py", line 149, in _dict_to_list
ValueError: dict contains fields not in fieldnames: 'timestamp'
taers232c commented 2 years ago

Jay,

Congratulations on being the first user to try this combination. For having to suffer, you get to choose where the timestamp column goes.

I have very limited Internet access, this may not get fixed until Sunday evening.

Ross

On Sat, Apr 16, 2022 at 1:31 PM Jay @.***> wrote:

Hi Ross!!

I've noticed that csv_output_timestamp_column works as expected:

@.***:~$ gam config csv_output_timestamp_column timestamp print domains domainName,parentDomainName,creationTime,type,verified,timestamp xxx,xxx,xxx,xxx,xxx,2022-04-16T20:11:00+00:00 yyy,yyy,yyy,yyy,yyy,2022-04-16T20:11:00+00:00

However, when combined with formatjson gam fails:

@.***:~$ gam config csv_output_timestamp_column timestamp print domains formatjson Traceback (most recent call last): File "init.py", line 61974, in ProcessGAMCommand File "init.py", line 13720, in doPrintShowDomains File "init.py", line 7584, in writeCSVfile File "init.py", line 7297, in writeCSVToStdout File "init.py", line 7275, in writeCSVData File "csv.py", line 157, in writerows File "csv.py", line 149, in _dict_to_list ValueError: dict contains fields not in fieldnames: 'timestamp'

I've tried several gam commands randomly and all of them exhibit the same behaviour so I think this might be affecting all of them:

@.***:~$ gam config csv_output_timestamp_column timestamp print users formatjson Getting all Users, may take some time on a large Google Workspace Account... Got x Users: a - b Traceback (most recent call last): File "init.py", line 61974, in ProcessGAMCommand File "init.py", line 37479, in doPrintUsers File "init.py", line 7584, in writeCSVfile File "init.py", line 7297, in writeCSVToStdout File "init.py", line 7275, in writeCSVData File "csv.py", line 157, in writerows File "csv.py", line 149, in _dict_to_list ValueError: dict contains fields not in fieldnames: 'timestamp'

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/262, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL3IHNXHDEJU6LXLL7LVFMPR5ANCNFSM5TS2VSJA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Ross Scroggs @.***

taers232c commented 2 years ago

Jay,

I decided to put the timestamp column right before JSON. Fixed in 6.20.06

Ross

On Sat, Apr 16, 2022 at 8:10 PM Ross Scroggs @.***> wrote:

Jay,

Congratulations on being the first user to try this combination. For having to suffer, you get to choose where the timestamp column goes.

  • At the end, before JSON, somewhere else?

I have very limited Internet access, this may not get fixed until Sunday evening.

Ross

On Sat, Apr 16, 2022 at 1:31 PM Jay @.***> wrote:

Hi Ross!!

I've noticed that csv_output_timestamp_column works as expected:

@.***:~$ gam config csv_output_timestamp_column timestamp print domains domainName,parentDomainName,creationTime,type,verified,timestamp xxx,xxx,xxx,xxx,xxx,2022-04-16T20:11:00+00:00 yyy,yyy,yyy,yyy,yyy,2022-04-16T20:11:00+00:00

However, when combined with formatjson gam fails:

@.***:~$ gam config csv_output_timestamp_column timestamp print domains formatjson Traceback (most recent call last): File "init.py", line 61974, in ProcessGAMCommand File "init.py", line 13720, in doPrintShowDomains File "init.py", line 7584, in writeCSVfile File "init.py", line 7297, in writeCSVToStdout File "init.py", line 7275, in writeCSVData File "csv.py", line 157, in writerows File "csv.py", line 149, in _dict_to_list ValueError: dict contains fields not in fieldnames: 'timestamp'

I've tried several gam commands randomly and all of them exhibit the same behaviour so I think this might be affecting all of them:

@.***:~$ gam config csv_output_timestamp_column timestamp print users formatjson Getting all Users, may take some time on a large Google Workspace Account... Got x Users: a - b Traceback (most recent call last): File "init.py", line 61974, in ProcessGAMCommand File "init.py", line 37479, in doPrintUsers File "init.py", line 7584, in writeCSVfile File "init.py", line 7297, in writeCSVToStdout File "init.py", line 7275, in writeCSVData File "csv.py", line 157, in writerows File "csv.py", line 149, in _dict_to_list ValueError: dict contains fields not in fieldnames: 'timestamp'

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/262, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL3IHNXHDEJU6LXLL7LVFMPR5ANCNFSM5TS2VSJA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Ross Scroggs @.***

-- Ross Scroggs @.***

jay-eleven commented 2 years ago

Ross,

I think slapping the timestamp column before the JSON fields makes the most sense as it would mimic a typical log file.

I've tested a few gam commands and all of them work perfectly now. Awesome!! I was expecting the timestamp to be INSIDE the JSON like any other field, but I guess a man can't have everything... 🤷🏻‍♂

Documentation doesn't explain how the timestamp is formatted:

...
csv_output_timestamp_column
        Name of column in the CSV output file to contain a timestamp
        Default: ''
...

Source: wiki/gam.cfg#variables

According to my (limited) tests, formatting seems to be ISO 8601: YYYY-MM-DDTHH:MM:SS<TZ>. I always get <TZ>=+00:00, suggesting timestamp is always UTC. Can you please confirm and I'll update GAM config wiki page:

csv_output_timestamp_column
        Name of column in the CSV output file to contain a timestamp.
        Format is ISO8601 with UTZ timezone: YYYY-MM-DDTHH:MM:SS±hh:mm.
        Eg: 2022-05-11T11:46:57+00:00.
        Default: ''
taers232c commented 2 years ago

Jay.,

The timestamp should be expressed in the gam.cfg todrive_timezone = '' if it's set, otherwise gam.cfg timezone

Ross

On Sun, Apr 17, 2022 at 9:04 AM Jay @.***> wrote:

Ross,

I think slapping the timestamp column before the JSON fields makes the most sense as it would mimic a typical log file.

I've tested a few gam commands and all of them work perfectly now. Awesome!!

Documentation doesn't explain how the timestamp is formatted:

... csv_output_timestamp_column Name of column in the CSV output file to contain a timestamp Default: '' ...

Source: wiki/gam.cfg#variables https://github.com/taers232c/GAMADV-XTD3/wiki/gam.cfg#variables

According to my (limited) tests, formatting seems to be ISO 8601: YYYY-MM-DDTHH:MM:SS. I always get TZ=+00:00, suggesting timestamp is always UTC. Can you please confirm and I'll update GAM config page https://github.com/taers232c/GAMADV-XTD3/wiki/gam.cfg#variables.

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/262#issuecomment-1100907070, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL47O5HZJ6KQ7YHAJJTVFQZAVANCNFSM5TS2VSJA . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

taers232c commented 2 years ago

Jay,

todrive_timezone not involved, just gam.cfg timezone, format is todrive_timeformat, see: https://github.com/taers232c/GAMADV-XTD3/wiki/gam.cfg#variables

Ross

On Sun, Apr 17, 2022 at 1:24 PM Ross Scroggs @.***> wrote:

Jay.,

The timestamp should be expressed in the gam.cfg todrive_timezone = '' if it's set, otherwise gam.cfg timezone

Ross

On Sun, Apr 17, 2022 at 9:04 AM Jay @.***> wrote:

Ross,

I think slapping the timestamp column before the JSON fields makes the most sense as it would mimic a typical log file.

I've tested a few gam commands and all of them work perfectly now. Awesome!!

Documentation doesn't explain how the timestamp is formatted:

... csv_output_timestamp_column Name of column in the CSV output file to contain a timestamp Default: '' ...

Source: wiki/gam.cfg#variables https://github.com/taers232c/GAMADV-XTD3/wiki/gam.cfg#variables

According to my (limited) tests, formatting seems to be ISO 8601: YYYY-MM-DDTHH:MM:SS. I always get TZ=+00:00, suggesting timestamp is always UTC. Can you please confirm and I'll update GAM config page https://github.com/taers232c/GAMADV-XTD3/wiki/gam.cfg#variables.

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/262#issuecomment-1100907070, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL47O5HZJ6KQ7YHAJJTVFQZAVANCNFSM5TS2VSJA . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

-- Ross Scroggs @.***

jay-eleven commented 2 years ago

Got it. Thanks for the clarification! Closing...