Cannot get_all_db_data

[benchub]

$ python manage.py get_all_db_data
Object of type 'datetime' is not JSON serializable
Traceback (most recent call last):
  File "/home/pygmy/pygmy/engine/management/commands/get_all_db_data.py", line 16, in handle
    AWSData().describe_ec2_instances()
  File "/home/pygmy/pygmy/engine/aws_wrapper.py", line 175, in describe_ec2_instances
    self.save_ec2_data(instance)
  File "/home/pygmy/pygmy/engine/aws_wrapper.py", line 237, in save_ec2_data
    print(json.dumps(instance, indent=4))
  File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.6/json/encoder.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.6/json/encoder.py", line 437, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'datetime' is not JSON serializable

[denishpatel]

@AadiMehta would you please working with @benchub to make sure he can setup environment properly?

[AadiMehta]

@benchub This is just a print statement error. Can you please take latest pull. That should resolve this

[benchub]

The lastest pull makes this run without errors... but from what I can see, nothing happens. (engine_instancestateinfo is empty after running.) Is there a way to run it with debug output?

Alternatively I'd be happy to do a screensharing session if that works for you.

[benchub]

OK, so looking to this more.....

I am successfully pulling instances from ec2. \o/

Sadly, I am unable to psql to any of those instances. It kept failing like so:

FATAL:  Peer authentication failed for user "pygmy"
Traceback (most recent call last):
  File "/home/pygmy/pygmy/engine/postgres_wrapper.py", line 13, in __init__
    self.conn = psycopg2.connect(host=DB_HOST, database=DB_NAME, user=DB_USER, password=DB_PASS, port=DB_PORT)

I assumed this was due to hba errors, but hba seemed correct, and double-checking postgres logs, it turned out that pygmy wasn't even attempting to log in. I threw in a debug line to print out the variables pygmy was using to connect to each instance; it turns out for every instance, pygmy is using: host=, database=postgres, user=pygmy, password=pygmy, port=5432

That is surprising to me, given that engine_dbcredentials has the actual password for pygmy to use (which is not "pygmy", although the role is). And, you know, how the host is completely unset.

[benchub]

Trying to track this down, this seems due to the line

            db_conn = PostgresData(instance.publicDnsName, "pygmy", "pygmy", "postgres")

on engine/management/commands/get_all_db_data.py:34, which seems quite clearly incorrect.

1 - instance.publicDnsName should be instance.privateIpAddress 2 - user/role should not be hardcoded to pygmy.

[amolvagare]

• user/role is taking from secrets pages for RDS and EC2 separatly
• for now keeping publicDnsName to test it from local

we will change it to privateIpAddress once code is stable

[amolvagare]

main issue of command failure is not present hence we can close it

[AadiMehta]

SHA-1: 2fb166f

Changes: Populate settings now ask for which regions to enable sync of Populate settings can now configure secrets Populate settings can now enable/disable RDS/EC2 sync Case correction Pygmy now using the right db credentials