Data availability less than GH apis provide, charted data timescale is off

[titpetric]

Description:

Reviewing data for longer intervals surfaces issue in the chart and missing data

Steps to reproduce:

1. set a 90 day interval
2. inspect pull requests insight chart

Expected behavior:

• Expected the chart to be aligned with the interval,
• Expected 90 days of data

Actual behavior:

Graph x axis is showing too large of interval Data beyond 30 days not available

GH action details should be available via GH apis for something like a 13 month interval, last time I checked. I'm unsure if any background sync process has failed or is in progress, as there are no UI details for it

[jayantbh]

Hey, thanks for filing this one! Currently the data is only backfilled upto a month in the past.

It should be easy enough on a system level to simply backfill data for longer. This will be handled very soon. :)

The chart actually shows a time interval double what you selected so you're able to see a change compared to the previous interval of equal duration. But yes, perhaps that could be communicated better compared to today.

[titpetric]

I'd definitely like to backfill this to a longer interval, up to whatever GH actions provide. It feels bad that 90%+ of the data is missing, definitely did not get the value of comparisons, longer intervals to see any existing trends...

[jayantbh]

While we handle this, I'm super glad for how involved you've been in sharing feedback on this. :) Thanks! 🚀

Of course, some of this stuff is pretty straight-forward to sort out too, so if you or someone on your team would like to lend us a hand, that would be awesome! 🕺

And finally, did you know that we have a separate cloud offering that does support 3 month backfills out of the box and a lot more integrations and features?

It's totally optional, but if you're interested you can sign up for it here: https://app.middlewarehq.com

[titpetric]

Best I can do at this time is see if golang is close to typescript and help out with some BE implementation details :)

[jayantbh]

The backend part of the codebase is in python, but it should feel somewhat similar to JavaScript in how in works in many areas. 😄

[titpetric]

Then no. 🤣 Respectfully, indentation based languages are not my jam.

On Wed, May 22, 2024, 18:19 Jayant Bhawal @.***> wrote:

The backend part of the codebase is in python, but it should feel somewhat similar to JavaScript in how in works in many areas. 😄

— Reply to this email directly, view it on GitHub https://github.com/middlewarehq/middleware/issues/371#issuecomment-2125200294, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABY7ECEXACPEHW677UQEGLZDTAQVAVCNFSM6AAAAABH7XFOSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRVGIYDAMRZGQ . You are receiving this because you authored the thread.Message ID: @.***>

[samad-yar-khan]

Hey @titpetric,

We have released a fix to assist you with data backfilling. We've introduced the DEFAULT_SYNC_DAYS environment variable, which allows you to specify how much data to sync during the initial synchronization. To use this feature, add DEFAULT_SYNC_DAYS=90 to your .env file and then try using the app again.

Here are the steps:

1)Pull the latest version of the repository. 2)Delete existing Docker volumes (this will delete your data).

3)Set DEFAULT_SYNC_DAYS to the number of days you wish to backfill. (Note: A higher number of days will result in a longer synchronization time.)

4)Run the app.

We will soon add a feature to allow on-demand data backfilling.

[jayantbh]

Hey all, closing this issue assuming the above workaround worked. Feel free to reply to this thread in case it didn't work.