search

carbonblack / cbapi-python

Carbon Black API - Python language bindings
Other
147 stars 86 forks source link

Iterating over binaries throws an exception: object mutated #40

Closed adamsmutnicki closed 8 years ago

adamsmutnicki commented 8 years ago

I have a script which collects information about all or most of binaries we have in CB instance. Its takes a while to download all of them, but I was never able to finish the full cycle, as iterating over binaries always thrown an exception "Object mutated", but in different places:

Example 1: File "/root/git/CB-ATA/new/cb_from_server_feed_binaries.py", line 52, in for e in b.endpoints: File "/usr/local/lib/python3.5/dist-packages/cbapi/response/models.py", line 779, in endpoints for endpoint in endpoint_list] File "/usr/local/lib/python3.5/dist-packages/cbapi/response/models.py", line 779, in for endpoint in endpoint_list] File "/usr/local/lib/python3.5/dist-packages/cbapi/connection.py", line 239, in select return select_instance(self, cls, unique_id, _args, _kwargs) File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 254, in call return self.cache[key] File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 42, in withlock return func(self, _args, _kwargs) File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 175, in getitem self.cleanup() File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 42, in withlock return func(self, _args, *_kwargs) File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 192, in cleanup for k in self.__expire_times: RuntimeError: OrderedDict mutated during iteration

Example 2: for b in data: File "/usr/local/lib/python3.5/dist-packages/cbapi/query.py", line 187, in _perform_query yield self._doc_class.new_object(self._cb, item) File "/usr/local/lib/python3.5/dist-packages/cbapi/response/models.py", line 714, in new_object return cb.select(Binary, item['md5'], initial_data=item) File "/usr/local/lib/python3.5/dist-packages/cbapi/connection.py", line 239, in select return select_instance(self, cls, unique_id, _args, _kwargs) File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 257, in call self.cache[key] = value File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 42, in withlock return func(self, _args, _kwargs) File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 168, in setitem self.cleanup() File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 42, in withlock return func(self, _args, *_kwargs) File "/usr/local/lib/python3.5/dist-packages/cbapi/cache/lru.py", line 192, in cleanup for k in self.__expire_times: RuntimeError: OrderedDict mutated during iteration

Code which runs this: c = CbEnterpriseResponseAPI(profile="default") data=c.select(Binary) for b in data:

do something, skipped

[...]

get endpoints list for a given binary

endpoints_list=[] for e in b.endpoints:

do something, skipped.

Can it be easily fixed or circumvent? Adam

jgarman commented 8 years ago

There is an issue with the LRU library I use for caching and Python 3.5 - this error is new with Python 3. I will take a look

jgarman commented 8 years ago

Try replacing your cache/lru.py file with the one from this branch: https://github.com/carbonblack/cbapi-python/blob/fix-python3-lru-cache/src/cbapi/cache/lru.py. It will be rolled into a future release once I've had a chance to properly test it.

adamsmutnicki commented 8 years ago

Hi Jason, Just run a test and it works without any problems. I've also done a test with Python2 version of this scripts and it works without any problems. So I'll waiting for a release of this patch. Regards Adam