Fixed iterators to handle daily throttling and interval throttling

BenFidge commented 5 years ago

@kevinobee This was discussed previously. Each iterator has subtle differences in how they page through results, hence apparent repetition but really isn't as there's no one size fits all for this. The logic to determine API usage and whether to retry is wrapped up in the base class, so its just try..catch logic added to each class in this which isnt really repetition

kevinobee commented 5 years ago

@BenFidge are you aware that the integration tests are failing in the repos? Trying to determine whether the failures are related to changes in this PR. Any views on that?

dotnet test Crawling.Hubspot.sln --filter Integration

example failure:

  X Crawling.HubSpot.Integration.Test.DataIngestion.CorrectNumberOfEntityTypes(entityType: "/Provider/Root", expectedCount: 1) [1ms]
  Error Message:
   System.AggregateException : One or more errors occurred.
---- System.InvalidOperationException : Communication to HubSpot unavailable. Request to https://api.hubapi.com/ integrations/v1/limit/daily failed, response  (NotFound)
---- The following constructor parameters did not have matching fixture data: HubSpotTestFixture fixture
  Stack Trace:

----- Inner Stack Trace #1 (System.InvalidOperationException) -----
   at CluedIn.Crawling.HubSpot.Crawler.GetData(CrawlJobData jobData) in C:\Users\kevin\Crawling.Hubspot\src\HubSpot.Crawling\Crawler.cs:line 35
   at CluedIn.Crawling.CrawlerRunner`1.Run(IAgentJobProcessorState`1 state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Reactive.Stubs.<.cctor>b__1(Exception ex)
   at System.Reactive.ObserverBase`1.OnError(Exception error)
   at CluedIn.Crawling.CrawlerRunner`1.Run(IAgentJobProcessorState`1 state)
   at CluedIn.Crawling.DebugCrawlerHost`1.Execute(Dictionary`2 credentials, Guid providerId)
   at Crawling.HubSpot.Integration.Test.HubSpotTestFixture..ctor() in C:\Users\kevin\Crawling.Hubspot\test\integration\Crawling.HubSpot.Integration.Test\HubspotTestFixture.cs:line 37
----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----

  X Crawling.HubSpot.Integration.Test.DataIngestion.CorrectNumberOfEntityTypes(entityType: "/PhoneCall", expectedCount: 3) [1ms]
  Error Message:
   System.AggregateException : One or more errors occurred.
---- System.InvalidOperationException : Communication to HubSpot unavailable. Request to https://api.hubapi.com/ integrations/v1/limit/daily failed, response  (NotFound)
---- The following constructor parameters did not have matching fixture data: HubSpotTestFixture fixture
  Stack Trace:

----- Inner Stack Trace #1 (System.InvalidOperationException) -----
   at CluedIn.Crawling.HubSpot.Crawler.GetData(CrawlJobData jobData) in C:\Users\kevin\Crawling.Hubspot\src\HubSpot.Crawling\Crawler.cs:line 35

BenFidge commented 5 years ago

@kevinobee Merging so can fix naming conflict.

BenFidge commented 5 years ago

@kevinobee Merging so i can fix naming conflict

CluedIn-io / CluedIn.Crawling.HubSpot

Fixed iterators to handle daily throttling and interval throttling #21