Feature request: consolidate config

[javydekoning]

It would be great if the config was stored in a single .json file and if the interactions with that file where to be moved to a separate module. This would make it easier to port bunq2ynab to another environment (For example AWS Lambda) because we'd only have to change to config.py module.

This would also clean up the code a lot (get rid of all the file reads/writes).

I'd imagine the config file to look something like this:

{
  "bunq": {
    "api_token":"",
    "priv_key":"",
    "install_token":"",
    "server_pub_key":"",
    "session_token":""
  },
  "bunq2ynab": [
    {
      "bunq_user":"",
      "bunq_acc":"",
      "ynab_budget":"",
      "ynab_acc":""
    },
    {
      "bunq_user":"",
      "bunq_acc":"",
      "ynab_budget":"",
      "ynab_acc":""
    }
  ],
  "ynab": {
    "accesstoken":"",
    "clientid":"",
    "clientsecret":""
  }
}

This would also allow users to setup multiple bunq acc <-> ynab budget pairs to sync.

An example of how the configuration module could look:

from logger import configure_logger
LOGGER = configure_logger(__name__)

class configuration: 
    """Class used for loading configuration from different sources.
    """

    def __init__(self, location):
        self.location = location
        l = location.split(':')
        try:
            self.type = l[0]
            self.path = l[1]
            LOGGER.debug('Using {0} with path {1}'.format(self.type, self.path))
        except: 
            LOGGER.debug('Failed to parse location. Expecting "type:path", e.g. "file:./config.json"')
            raise

    def load(self):
        LOGGER.info(self.location)
        if (self.type == 'file'):
            return self.load_file()

    def load_file(self):
        with open(self.path, 'r') as f:
            data = f.read()
            return json.loads(data)

This would allow us to also load config from Systems manager for example by adding:

    def load_ssm(self):
         parameter_store = ParameterStore()
         resp = parameter_store.fetch_parameter(self.path)
         return resp

[wesselt]

Thanks for the suggestion!

Both the bunq and ynab account names are command-line parameters. So you can already sync multiple accounts. For example I've got 2 copies of the autosync script running. This sets up one callback per account.

I'll look into using a JSON file for storage.

[javydekoning]

I forked and implemented the above (amongst other things) here: https://github.com/javydekoning/bunq-ynab-aws-lambda/tree/master/sam-app/bunq2ynab

If any of it is useful let me know.

[wesselt]

Cool, I have some time this weekend, I'll have a look if I can merge it!

[wesselt]

Finally managed to write code that automatically categorizes ZeroFX transactions. I'm now trying to consolidate the configuration on branch "config". Coming weeks I'll try to support multiple account pairs.

How can a Lambda Python script read the JSON that's posted to the endpoint?

[javydekoning]

I'm storing the config in Systems Manager Parameter store, see these lines for details: https://github.com/javydekoning/bunq-ynab-aws-lambda/blob/master/bunq2ynab/config.py#L40-L43 there is no posting of config files to the end-point.

I recently wrapped up my fork of your older version into a SAM app and published it in the Serverless Application Repository. This allows users to deploy it, and only paste in their API secrets via the GUI wizard. The application template is here

If you managed to separate out to config I'd be happy to work with you on moving your version there instead. Also happy to walk you over how it works via a video call if you are interested.

[wesselt]

The SAM app looks good!

Thanks for pointing at the parameter store code, interesting. I assumed the Lambda would run whenever bunq sends a callback. The body of the callback is a json that tells you which account was changed. That could be useful to synchronize multiple accounts.

Would be happy to talk, how does Wednesday suit you?

[javydekoning]

A callback would be the best way to implement this. I never implemented that for bunq, but can certainly help there. Wednesday is fine by me. Can you drop me an email and timeslot at

[wesselt]

Thanks for the explanation on Wednesday! Using the Python logger was some more work than I expected. You can't log in the config before you have a logger, and you can't configure the logger before you have the config. Now I just start with the default logger and "upgrade" it when the configuration is loaded.

The "-d" or "--detailed" command line switch now uses the more detailed logging format.

The configuration is consolidated in "config.json" (the application only reads this.) The state (basically bunq tokens) is stored in "state.json", which is both read and write.

You can specify multiple accounts to sync in "config.json".

{
    "api_token": "...",
    "personal_access_token": "...",
    "accounts": [{
        "bunq_user_name": "*",
        "bunq_account_name": "*",
        "ynab_budget_name": "yourbudget",
        "ynab_account_name": "*"
    }, {
        "bunq_account_name": "account2",
        "ynab_budget_name": "anotherbudget",
        "ynab_account_name": "second_account"
    }],
    "single_ip": "True"
}

If you use * or omit anything, it will match the accounts by name.

Now that the logging is there, what's the next thing for supporting Lambda?

[javydekoning]

Hi Wessel,

Great! Some optional feedback, do with it as you see fit :).

Using the Python logger was some more work than I expected. You can't log in the config before you have a logger, and you can't configure the logger before you have the config.

Are you saying because you need to load the log-level from the config-file? When you want to configure logging for your project, you should do it as soon as possible when the program starts. The way you can address this is by having a default value that can be overridden by environment variable (env is usually used Lambda, docker-compose files etc). Only after that should you read from the config file. So:

1. If LOG_LEVEL environment variable exists use it.
2. Otherwise initialize the default 'INFO'

logger.setLevel(os.environ.get("LOG_LEVEL", logging.INFO))

3. Next read the config file (you now already have logging). If loglevel is in the config file it should override the default only if LOG_LEVEL environment variable does not exist.

So in order of priority env > configfile > default

---

The "-d" or "--detailed" command line switch now uses the more detailed logging format.

This becomes confusing to me, you now have 4 settings that control logging:

"-d", "--detailed" (Changes log message format)
"-v", "--verbose" (Log content of JSON messages)
"-vv", "--headers" (Log HTTP headers)
loglevel (INFO, DEBUG, etc)

• -d Is there any reason someone would pick the 'old' log format? Why not make the new format the default?
• -v and -vv. I would remove. When the loglevel is set to DEBUG you can log HTTP headers and JSON objects.

These changes would make the logging experience far easier.

---

Now that the logging is there, what's the next thing for supporting Lambda?

A few things come to mind:

1. Support SSM for the config location (read from Systems Manager Parameter store instead of file). I can do this.
2. Create a SAM template and publish to SAR. (I can do this as well).
3. Make minor changes to bunq2ynab.py, Lambda requires an entry point like:

def handler(event, context):
  config.load()
  sync = sync.Sync()
  sync.populate()
  sync.synchronize()

# If running locally:
if (NOT RUNNING IN AWS LAMBDA):
  handler({},{})

[wesselt]

Hi Javy,

Thanks for the feedback! I changed the logging to what you suggested, except -v still exists as a shortcut to --log-level debug. The priority is now: parameter > environment > config.json > default.

3. Make minor changes to bunq2ynab.py, Lambda requires an entry point like:

I've added a file called aws_lambda.py that contains a handler function.

1. Support SSM for the config location (read from Systems Manager Parameter store instead of file). I can do this.

Let's do this next! What do you think about making a pull request that does this?

[javydekoning]

Thanks Wessel! I'll dedicate some time to work on this next week :)

[javydekoning]

• I've opened a PR with minor changes for the requirements.txt and the .gitignore

Support SSM for the config location (read from Systems Manager Parameter store instead of file)

This I've almost finished. Have it running, but need to add some extra error handling. I'll open a PR for that later.

I've added a file called aws_lambda.py that contains a handler function.

I don't think we actually need this, after the above 2 PR's are merged I'll work on minor edits to bunq2ynab so it will run on both AWS Lambda and local machine :).

[javydekoning]

Finished Lambda compatibility. If you want to have a preview look at this commit: https://github.com/wesselt/bunq2ynab/commit/38607aefd4b9246ce23fe271f2aa062c3dc3ac9a

Final step is to update the README file and test it for a day, I think I will be able to finalize everything this weekend.

[javydekoning]

I think we can close this, agreed @wesselt ?

[wesselt]

Yeah, I'll close it!