search

panoratech / Panora

One API to connect your data sources to your LLMs
https://docs.panora.dev
Apache License 2.0
867 stars 181 forks source link

Handle Pagination For Gitlab Connector. #428

Closed rajeshj11 closed 4 months ago

rajeshj11 commented 4 months ago

Describe the bug While I was addressing bounty ticket #367 , it seems that @mit-27 completed it ahead of me. However, it appears that some changes related to pagination were missed during the initial completion. Since I've already invested time in understanding and implementing pagination for this ticket, I'd like to review and address any necessary fixes if needed, to ensure that the solution meets the project requirements thoroughly.

screenshots Screenshot from 2024-05-14 21-54-24

Additional context

  1. Gitlab integration needs pagination handling.
  2. While syncing it may encounter http status 500 error as we hit the API multiple times. so it needs retry mechanism.

https://docs.gitlab.com/ee/api/rest/index.html#pagination

rajeshj11 commented 4 months ago

@rflihxyz ^^ Please assign it to me if it is a valid issue.

naelob commented 4 months ago

@rajeshj11 Ok cool ! work on it considering @mit-27 's work and I'll compensate as well !

rajeshj11 commented 4 months ago

Thanks , will raise pr and let you know

On Wed, 15 May 2024, 02:19 nael, @.***> wrote:

@rajeshj11 https://github.com/rajeshj11 Ok cool ! work on it considering @mit-27 https://github.com/mit-27 's work and I'll compensate as well !

— Reply to this email directly, view it on GitHub https://github.com/panoratech/Panora/issues/428#issuecomment-2111116260, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJHVFCHHW2GGTWZDBN7VMV3ZCJ2HPAVCNFSM6AAAAABHWQ2ZGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJRGEYTMMRWGA . You are receiving this because you were mentioned.Message ID: @.***>

rajeshj11 commented 4 months ago

@naelob please review https://github.com/panoratech/Panora/pull/429

rflihxyz commented 4 months ago

Hello, @rajeshj11

Thanks for that contribution! We really do appreciate that. We're currently evaluating multiple types of pagination for our API : cursor-based, and offset based. We haven't made a decision yet about which approach to take, for that reason, we can not accept this contribution immediately.

Do you want us to update you once we move to that feature? I just created this issue to discuss it.