Defaults in values.yml for fakecega

wandergeek commented 5 years ago

Description

I'm trying to get local versions of CEGA and LEGA chatting with one another and having a hard time wrangling the config. What variables in values.yml (for fakecega and lega) need to be changed?

As of now, my values.yml looks like so:

config:
  log: "debug"
  broker_connection_attempts: 30
  broker_enable_ssl: "false"
  broker_heartbeat: 0
  broker_host: "lega-localega-mq" 
  broker_port: 5672
  broker_retry_delay: 10
  broker_username: "guest" # local broker user
  broker_vhost: "/"
  cega_users_host: http://cega-users
  cega_users_endpoint: "/lega/v1/legas/users/%s?idType=username" # "/lega/v1/legas/users/%s?idType=username"
  cega_endpoint_json: "response.result"
  cega_mq_host: "cega-mq" # FQDN to cega mq host
  cega_vhost: "/"
  cega_port: 5672
  cega_username: "cega" # cega user
  keyserver_host: "lega-localega-keys" # defaults to localega-keys
  keyserver_endpoint: "/keys/retrieve/%s/private/bin?idFormat=hex"
  postgres_db_name: "lega"
  postgres_db_schema: "local_ega"
  postgres_host: "lega-localega-db" # defaults to localega-db
  postgres_try: 30
  postgres_sslmode: "prefer"
  postgres_user: "lega"
  res_host: "lega-localega-res" # defaults to localega-res
  filedatabase_host: "lega-localega-filedatabase" # defaults to localega-filedatabase
  dataedge_host: "lega-localega-dataedge" # defaults to localega-dataedge
  data_storage_type: "S3Storage" # S3Storage or FileStorege
  data_storage_url: "https://umccr-localega-dev.s3.amazonaws.com" # URl to S3 instance
  data_storage_s3_bucket: "umccr-localega-dev"
  data_storage_s3_region: "ap-southeast-2"
  data_storage_s3_chunk_size: 4 # Chunk size in MB
  data_storage_location: "/ega/data_archive" # path to data archive volume
  data_storage_mode: 2750

persistence: 
  enabled: true

secrets:
  cega_creds: "cega"
  cega_mq_pass: "lega"
  pgp_passphrase: "guest"
  shared_pgp_password: "guest"
  mq_password: "lega"
  postgres_password: "lega"
  s3_access_key: "xxxx"
  s3_secret_key: "xxxxx"

I have a helm installation of fakecega running as cega and am able to resolve these names.

A few other issues I've run into:

There are hardcoded hashes in cega/conf/cega.json what values do these correspond to? Do these have anything to do with what's in values.yml?
I'm also unsure what cega_creds cega_mq_pass mq_password refer to since they are undocumented in the readme.
What does the dummy user password hash in dummy.yml correspond to?
Should I be using values from trace.yml? Can we use these programmatically?

Proposed solution

Document default/hardcoded hashes or generate them on the fly
Document cega_creds cega_mq_pass mq_password
A simple procedure to spin up CEGA and LEGA together such that the defaults work -- ie if cega and lega are spun up with particular names in helm it "just works".

Definition of Done

Documentation and defaults updated

Great work on this by the way! I'm looking forward to getting these components singing! 🚀

blankdots commented 5 years ago

@wandergeek

we have an utility that generates them it is in https://github.com/NBISweden/LocalEGA-deploy-init and there is pending PR to make it easier; it is also part of this installation guide (see Readme)

When deploying a dev environment for the first time you need to create the secrets using the deploy.py script from LocalEGA-deploy-k8s.
Maybe this helps:
- cega_creds - are the User credentials used in Inbox: https://github.com/EGA-archive/EGA-auth/blob/master/auth.conf.sample#L8 for CEGA User REST Authentication see:
- https://github.com/NBISweden/LocalEGA-auth/blob/master/src/cega.c#L68
- or the Apache Mina Inbox: https://github.com/NBISweden/LocalEGA-inbox/blob/master/src/main/java/se/nbis/lega/inbox/sftp/CredentialsProvider.java#L43
- cega_mq_pass - CEGA MQ password to connect to that broker
- mq_password - LocalEGA MQ password

Maybe for the rest @jbygdell can help a bit more with what they are

wandergeek commented 5 years ago

Hey @blankdots thanks for that.

How does this script differ from the other configuration generation script in https://github.com/NBISweden/LocalEGA-deploy-k8s? Maybe its worth deprecating one of them?

Can you comment on the hashes in cega/conf/cega.json and dummy.yml?

blankdots commented 5 years ago

@wandergeek https://github.com/NBISweden/LocalEGA-deploy-k8s has been renamed to https://github.com/NBISweden/LocalEGA-deploy-init ... PR will follow :D

The password hash in dummy.yml I think we are not using it and the same might go for the cega/conf/cega.json one.

wandergeek commented 5 years ago

This is sorta related, I guess, but another thing I'm after is the private key that corresponds to that dummy user. I tried to replace it with my own key, but had an issue with paramiko: paramiko.ssh_exception.SSHException: not a valid RSA private key file.

For your reference this is what my user config looks like:

---
username: dummy
password_hash: $1$b0YnuEsc$8METjSpQmprwFwpxL0x7f0
pubkey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHubGbeMJHs6iymY/KLWNc+fSWSIbr0P/VousmAcRtjbFHJxDX27bKvXl3bx+t8wx3ho8i7969nqBu8dxRGv4yODCfg9m1Uc9KVq1XAWEgeleHS5V5Mp+yknMXa7uCPNZB9VSqU2knUXK06s9U9fcmJU1BH7VVETxBSu/EyVVq5Hyu+yXTon4Nqrubu22ZMQFE6CJd9ETgXokRD7QMD3bCt31AhSpMgV22ktV+R7UUP8nA2zWr4QRPnwe1j7k7MXhw3tDJKFfOoVRWIN9U/FummX7udYUQ0KU1kGCd5f771bkDvYJt3FM+JyMuP4R6eQXXgLBGGN1vRVu6A+eAm5Pl nrclark@5690L-150616-M.local
uid: 1
gecos: dummy user

and my private key:

And this is how I'm running it, via your testing script: python3 test.py conf/test_file config.yaml

I have confirmed both keys match.

blankdots commented 5 years ago

@wandergeek I think paramiko might be the actual issue here: https://github.com/paramiko/paramiko/issues/1382

This might help when generating keys: ssh-keygen -m PEM -t rsa -b 4096 -C "your_email@example.com"

Look for this header:

-----BEGIN RSA PRIVATE KEY-----

wandergeek commented 5 years ago

That fixed it. Thanks for that.

In other news, I'm trying to debug the auth pam module, but am unable to get any logging messages out of it. I've recompiled it with make debug3 and set SYSLOG = true in the makefile.

Can you please assist?

wandergeek commented 5 years ago

Also can you please please tell me what the password hash for the dummy user corresponds to? I have no idea how to generate my own and I am trying to get inbox working.

jbygdell commented 5 years ago

Also can you please please tell me what the password hash for the dummy user corresponds to? I have no idea how to generate my own and I am trying to get inbox working.

The dummy password should be dummy, but it should not be needed as the sftp inbox works with key based auth.

jbygdell commented 5 years ago

I'm trying to debug the auth pam module, but am unable to get any logging messages out of it. I've recompiled it with make debug3 and set SYSLOG = true in the makefie

Recompiling with debug3 is usually enough.

Make sure that you are using the nbisweden/ega-inbox:m4 image.

One thing you can test from you inbox container is to do the user lookup manually: curl -u $CEGA_ENDPOINT_CREDS http://<YOUR FAKE CEGA USERS ENDPOINT>/lega/v1/legas/users/dummy?idType=username

wandergeek commented 5 years ago

Ah ha, ok, using the m4 image worked. The values.yml defaults to latest. I'll fix it up. Heads up key auth doesn't want to work in latest-- I provide the key, but it still asks for a password for some reason.

Thanks for your help.

blankdots commented 5 years ago

@wandergeek We are in the process of addressing issues and tags, for now the stable tags are as @jbygdell pointed out m4 and for the cscfi/images m4-alpine, these will change in the following weeks, also may I close this issue and/or PR ?

wandergeek commented 5 years ago

I'll give it another whirl with your changes, will reopen if necessary.

NBISweden / LocalEGA-helm