search

jpetrucciani / hubspot3

python3.6+ hubspot client based on hapipy, but modified to use the newer endpoints and non-legacy python
MIT License
147 stars 73 forks source link

Running tests not against the live HubSpot demo account #40

Open rheinwerk-mp opened 5 years ago

rheinwerk-mp commented 5 years ago

My colleagues and I a currently working on extending the contacts API (as mentioned also in https://github.com/jpetrucciani/hubspot3/issues/39).

The tests for this (and other) API rely on HubSpot's live demo account. This causes some problems. Mainly the tests are failing at the moment because the demo account has reached its daily limit. We need to wait until the next day, when the limit is reset, to run all tests, hoping is has not been exceeded again.

Furthermore the tests can fail when the live database does not contain the expected records. Two examples: When the database is flushed, all tests relying on the contact from the BASE_CONTACT would fail. And if there are less than 20 records in the database, testing the limit parameter in the test test_get_all will not fail but will not test anything useful either since there will always be less than 20 results, regardless of the parameter value.

As a solution we would suggest to mock the API calls and their responses in the current unit tests. We would include the necessary changes (at least for the contact API tests) in the following PR.

Let me know what do you think.

jpetrucciani commented 5 years ago

Thanks for bringing this issue up!

Yeah it'd probably be best to mock those all out - I had started on rebuilding the test suite and quickly ran into all sorts of problems trying to get how I wanted to test things working (including a nasty workaround that you've found in a couple different places for loading a record on boot to use for testing https://github.com/jpetrucciani/hubspot3/blob/6791c78e8ec722a012615ef3d9badc33fb68954f/hubspot3/test/test_contacts.py#L12-L14).

Good point on the test_get_all tests as well - there's a lot of the testing that I hadn't completely thought through when I started writing the tests!

I'd agree with you that mocking the calls would probably be best - I'm open to any methods of doing so that you guys are comfortable with!

jpetrucciani commented 5 years ago

I've also created a topic on the HubSpot developer forum about this, and asked if we could potentially have a different key specific for testing - I do feel that mocking it is fine, but not entirely optimal. If we could be hitting the real API during tests, that'd allow us to test for any changes that they've made as well.

rheinwerk-mp commented 5 years ago

Great that you are open changing the tests.

Regarding testing against the live API: Yes, but I would suggest to create a different set of tests (integration tests) for this purpose. The unit tests on the other hand would only test the integrity of the code itself.

jpetrucciani commented 5 years ago

Good point - that makes a lot of sense!

jpetrucciani commented 5 years ago

As of the latest commit/release 3.2.17, I've included my test account key in the tests, so we should be able to use that key for integration tests (so at least we can get the old tests back up and running on Travis!). I've also switched the CI to use tox across the three supported versions we've specified.

I'm still on board with also building out a bunch of unit tests for the library that use your mocking methods - I'll just start slowly writing out some new tests when I get some more time!