search

singer-io / tap-mambu

GNU Affero General Public License v3.0
14 stars 21 forks source link

readme

tap-mambu

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

Streams

branches (GET v2)

communications (POST v2)

centres (GET v2)

clients (GET v2)

credit_arrangements (GET v2)

custom_field_sets (GET v1)

deposit_accounts (POST v2)

cards (GET v2)

deposit_products (GET v1)

deposit_transactions (POST v2)

groups (GET v2)

loan_accounts (POST v2)

loan_products (GET v1)

loan_transactions (POST v2)

tasks (GET v2)

users (GET v2)

gl_accounts (GET v1)

gl_journal_entries (POST v1)

activities (GET v1)

index rate sources (GET v2)

installments (GET v2)

audit_trail (GET V1)

Quick Start

  1. Install

    Clone this repository, and then install using setup.py. We recommend using a virtualenv:

    virtualenv -p python3 venv
source venv/bin/activate
python setup.py install
OR
cd .../tap-mambu
pip install .

  2. Dependent libraries The following dependent libraries were installed.

    pip install singer-python
pip install singer-tools
pip install target-stitch
pip install target-json

  3. Create your tap's config.json file. The subdomain is everything before .mambu.com in the Mambu instance URL. For the URL: https://stitch.sandbox.mambu.com, the subdomain would be stitch.sandbox. Lookback window applies only to loan transactions stream.

    {
    "username": "YOUR_USERNAME",
    "password": "YOUR_PASSWORD",
    "apikey": "YOUR_APIKEY",
    "subdomain": "YOUR_SUBDOMAIN",
    "start_date": "2019-01-01T00:00:00Z",
    "lookback_window": 30,
    "user_agent": "tap-mambu <api_user_email@your_company.com>",
    "page_size": "500",
    "apikey_audit": "AUDIT_TRAIL_APIKEY",
    "window_size": 7
}

    Note: The window_size parameter defaults to 1 day, which may cause slowdowns in historical sync for streams utilizing multi-threaded implementation. Conversely, using a larger window_size could lead to potential out-of-memory issues. It is advisable to select an optimal window_size based on the start_date and volume of data to mitigate these concerns.

    Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

    {
    "currently_syncing": "tasks",
    "bookmarks": {
        "audit_trail":"2021-05-20T09:59:09.780213Z", 
        "branches": "2019-06-11T13:37:51Z",
        "communications": "2019-06-19T19:48:42Z",
        "centres": "2019-06-18T18:23:53Z",
        "credit_arrangements": "2019-06-19T19:48:45Z",
        "custom_field_sets": "2019-06-11T13:37:56Z",
        "deposit_accounts": "2019-06-19T19:48:47Z",
        "cards": "2019-06-18T18:23:58Z",
        "deposit_products": "2019-06-20T00:52:49Z",
        "deposit_transactions": "2019-06-20T00:52:40Z",
        "groups": "2019-06-19T19:48:41Z",
        "loan_accounts": "2019-06-11T13:37:52Z",
        "loan_products": "2019-06-20T00:52:43Z",
        "loan_transactions": "2019-06-19T19:48:44Z",
        "tasks": "2019-06-18T18:23:55Z",
        "users": "2019-06-20T00:52:46Z",
        "gl_journal_entries": "2019-11-01T00:00:00.000000Z",
        "gl_accounts": {
            "INCOME": "2019-07-03T15:15:43.000000Z",
            "EQUITY": "2019-07-03T15:15:43.000000Z",
            "LIABILITY": "2019-07-03T15:15:43.000000Z",
            "ASSET": "2019-07-03T15:15:43.000000Z",
            "EXPENSE": "2019-07-03T15:15:43.000000Z"
        }
    }
}

  4. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

    tap-mambu --config config.json --discover > catalog.json

    See the Singer docs on discovery mode here. The default catalog.json that gets generated does not have any selected streams. Therefore you must enable them manually in order to actually retrieve data. Each stream in the catalog.json that you want retrieved is expected to have a data field in the root metadata block:

    "selected" : true

    For example the following configuration will retrieve the branches stream:

    "stream": "branches",
"metadata": [
  {
    "breadcrumb": [],
    "metadata": {
      "selected" : true,
      "table-key-properties": [
        "id"
      ],
      "forced-replication-method": "INCREMENTAL",
      "valid-replication-keys": [
        "last_modified_date"
      ],
      "inclusion": "available"
    }
  }

  5. Run the Tap in Sync Mode (with catalog) and write out to state file

    For Sync mode:

    tap-mambu --config config.json --catalog catalog.json > state.json
tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    To load to json files to verify outputs:

    tap-mambu --config config.json --catalog catalog.json | target-json > state.json
tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    To pseudo-load to Stitch Import API with dry run:

    tap-mambu --config config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

  6. Test the Tap

    While developing the Mambu tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

    pylint tap_mambu -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments

    Pylint test resulted in the following score:

    Your code has been rated at 9.87/10.

    To check the tap and verify working:

    tap-mambu --config config.json --catalog catalog.json | singer-check-tap > state.json
tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    Check tap resulted in the following:

    The output is valid.
It contained 589 messages for 18 streams.

    24 schema messages
    503 record messages
    62 state messages

Details by stream:
+----------------------+---------+---------+
| stream               | records | schemas |
+----------------------+---------+---------+
| loan_products        | 2       | 1       |
| users                | 3       | 1       |
| gl_journal_entries   | 281     | 1       |
| branches             | 2       | 1       |
| groups               | 3       | 1       |
| credit_arrangements  | 2       | 1       |
| clients              | 104     | 1       |
| cards                | 1       | 3       |
| loan_accounts        | 7       | 1       |
| deposit_transactions | 32      | 1       |
| centres              | 2       | 1       |
| gl_accounts          | 18      | 5       |
| communications       | 1       | 1       |
| deposit_accounts     | 3       | 1       |
| tasks                | 8       | 1       |
| loan_transactions    | 9       | 1       |
| custom_field_sets    | 21      | 1       |
| deposit_products     | 4       | 1       |
+----------------------+---------+---------+

Copyright © 2019 Stitch