search

github-early-access / copilot-custom-models-beta

Feedback repo for the Limited Public Beta of Copilot Custom Models
3 stars 0 forks source link

Possibly Inaccurate Usage API Metrics, or Unclear Copilot Access Controls #6

Open wbond opened 2 weeks ago

wbond commented 2 weeks ago

We added 11 users to a custom org we've created to allow us to gate our new custom model. We set up three people to have Copilot through that org. Later yesterday we added a fourth.

We've also implemented a pipeline to pull data from the usage API so we can perform some data analysis internally.

Yesterday, before adding the fourth person to have Copilot access through the org, the usage API was telling us there were 7 active users.

How is this possible? Is it possible that all 11 members of the organization are automatically getting access to the custom model, even if they've only been granted to access to Copilot through another org in our enterprise?

thejoebourneidentity commented 2 weeks ago

@justincampbell @ivanpk , what do we need to triage this and investigate?

justincampbell commented 6 days ago

@wbond There was an issue at that time with the metrics we had stored for that timeframe. The issue has been resolved, and the data was backfilled. Can you verify that it looks correct now?

We also verified that only users with active Copilot seats (3) were getting the custom model, the rest were accessing the base Copilot model (through a different org as you said).

Let us know if you have any more questions, thanks!

wbond commented 6 days ago

@imarban Can you verify if our data looks good now?

imarban commented 6 days ago

Hey, thanks for the response @justincampbell. Couple of questions

The issue has been resolved, and the data was backfilled

Should I read this as in: if I consume the data for the days we were seeing issues before, the data should be fixed? Because, unfortunately I still see the API reports 7 active users for days where we only had three people.

For what is worth, I also see the issue happening for the last few days where the API returns 4 in the "total_active_users" field for Oct 1, even if we only had 2 people accessing Copilot through this new org.

justincampbell commented 5 days ago

@imarban Thanks for checking, we're looking into it again