search

GrantQuick / BlackbaudSkyApi-RENXT

A Power BI custom data connector for Blackbaud's Raiser's Edge NXT (SKY API)
28 stars 8 forks source link

Over 1 million gift records #5

Closed vitendy closed 5 years ago

vitendy commented 5 years ago

Is there a way to limit the number of gifts getting pulled? Power BI desktop seems to be choking on this, that is, load never finishes. Do you have recommendations how to work around this issue?

GrantQuick commented 5 years ago

Unfortunately no, not currently. Power BI won't allow data loads to take more than one hour. For large datasets like this (where the bottleneck is the API itself), my recommendation would be to build a local data warehouse and perform daily updates of new and changed records via calls to the API. That implementation would be out of scope for this connector though. Since there's no local data store, it's not possible to configure the connector to just bring back the latest records - it's all or nothing.

vitendy commented 5 years ago

Well, Power Query M is fun! I modified the gifts REST endpoint URL to include the date_added parameter. The query seems to respect this, so it brings back a limited number of gifts, but for some reason it gets stuck on the refresh dialog.

Fiddler shows the few requests going out to BB. Then the requests stop, but the PBID refresh screen keeps adding records. It's stuck in a loop.

If you have any clues why this is happening, let me know. I don't mean for you to debug this for me.

GrantQuick commented 5 years ago

To be honest, it's you helping me debug ;)

You've found a legit bug caused by an incorrect transformation in the Gift endpoint. I've fixed this and committed a new version. Let me know if this fixes it for you.

vitendy commented 5 years ago

I added date_added to the gifts endpoint url after downloading the updated code. No change in behavior... just keeps going, and going, ...

GrantQuick commented 5 years ago

Okay, I can't replicate that. When you trace through fiddler, do all the API calls complete?

Difficult for me to debug as I'm not experiencing the issue. However if you want you have a go at seeing if reduced datasets load into PBI, you could break this line by changing the "next_link" string to something else so that only one call will be made to the API, although it could be that you're already seeing that with the addition of the date_added param:

GetNextLink = (response) as nullable text => Record.FieldOrDefault(response, "next_link");

GrantQuick commented 5 years ago

Closing issue, cannot replicate.