Hackathon: chargebee source

[AstrakhantsevaAA]

Problems:

1. Auth was generated with username as a string, it should be a secret:

       source_config: RESTAPIConfig = {
           "client": {
               "base_url": base_url,
               "auth": {
                   "type": "http_basic",
                   "username": "username", # should be a variable 
                   "password": password,
               },
           },

   Funny fact: if I set username as a dlt secret, then dlt will take var ENV $USERNAME

2. Run pipeline: should be "offset_param": "next_offset",

3. Total in items issue:

   fixed with replacing in dlt.sources.helpers.rest_client.paginators.RangePaginator.update_state

               if total is None:
                   total = len(response_json["list"])
                   # self._handle_missing_total(response_json)

   full tracebacks you can find here https://www.notion.so/dlthub/Alena-Chargebee-e390697e31b44dd48b7afeee48096688?pvs=4

Highlights:

• Amazing logs, I can see what’s happening under the hood
• Picking endpoints mechanism is convenient (if there are not many of them)
• Autogenerated Readme with endpoints (resources) is a super feature!! I hope we will add more info here in the future

Notes:

• It would be cool if gitignore with .duckdb and secrets.toml was autogenerated too

• I would make this INFO more readable, add punctuation for example:

  2024-05-21 17:19:52,640|[INFO                 ]|507526|132218115059712|dlt|client.py|paginate:228|Paginator OffsetPaginator at 78405fe9dde0: current offset: 100 offset_param: offset_next limit: 100 total_path: total maximum_value: 20 does not have more pages

• It works super long when I'm picking all endpoints

[sh-rp]

Hey @AstrakhantsevaAA, thanks for your review! I have created two ticket for the generator here:

• https://github.com/orgs/dlt-hub/projects/12/views/1?pane=issue&itemId=63886018
• https://github.com/orgs/dlt-hub/projects/12/views/1?pane=issue&itemId=63886078

Did you create a ticket for the paginator in the main repo or should I do this too? And let me know if I overlooked anything that should get an additional ticket.

[sh-rp]

PS: Secrets toml is generated, maybe you used and older version of the generator.

[AstrakhantsevaAA]

@sh-rp Thank for tickets!

Did you create a ticket for the paginator in the main repo or should I do this too?

I didn't but I can do that.

And let me know if I overlooked anything

autogenerate gitignore file? our regular, with secret.toml and .duckdb

[AstrakhantsevaAA]

Could you also replace dashes "-" in source names, e.g. "chargebee-pipeline" to "chargebee_pipeline", I cannot import module with dashes in name. And add empty init.py in the "chargebee-pipeline" folder automatically

[sh-rp]

Could you also replace dashes "-" in source names, e.g. "chargebee-pipeline" to "chargebee_pipeline", I cannot import module with dashes in name. And add empty init.py in the "chargebee-pipeline" folder automatically this is coming with the next release

[sh-rp]

gitignore is now also created. and I check the secrets.toml has all the vars. if no auth is detected, it will still prepare a secrets toml with the right sections for later use.