Closed Shak2000 closed 3 years ago
When I tried to run the Washington DC script it failed. Have you tried running it?
Well it's still giving me problem, it runs for 5 mins and then fails for following error bson.errors.InvalidDocument: cannot encode object: 20032, of type: <class 'numpy.int64'>
. Below is the stack trace
Traceback (most recent call last):
File "c:/Users/madha/IdeaProjects/AddResources/AddResources/Washington_DC_Shelters/Washington_DC_Shelters.py", line 70, in <module>
dc_shelters_scraper.main_scraper(client)
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\shared_code\base_scraper.py", line 144, in main_scraper
dc = locate_potential_duplicate(
**File "c:\Users\madha\IdeaProjects\AddResources\AddResources\shared_code\utils.py", line 126, in locate_potential_duplicate
dupe_candidate = coll.find_one(**
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\collection.py", line 1319, in find_one
for result in cursor.limit(-1):
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\cursor.py", line 1207, in next
if len(self.__data) or self._refresh():
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\cursor.py", line 1124, in _refresh
self.__send_message(q)
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\cursor.py", line 999, in __send_message
response = client._run_operation_with_response(
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\mongo_client.py", line 1368, in _run_operation_with_response
return self._retryable_read(
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\mongo_client.py", line 1471, in _retryable_read
return func(session, server, sock_info, slave_ok)
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\mongo_client.py", line 1360, in _cmd
return server.run_operation_with_response(
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\server.py", line 101, in run_operation_with_response
message = operation.get_message(
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\message.py", line 343, in get_message
request_id, msg, size, _ = _op_msg(
File "c:\Users\madha\IdeaProjects\AddResources\AddResources\.venv\lib\site-packages\pymongo\message.py", line 714, in _op_msg
return _op_msg_uncompressed(
bson.errors.InvalidDocument: cannot encode object: 20032, of type: <class 'numpy.int64'>
You are not seeing it because you are running it on fresh shelter database which doesn't have services
collection against which scraper will try to do the partial match. You can copy services collection in your mongoDB database and try running the script.
I parsed through the data set on food banks in British Columbia and the data set on homeless shelters in Washington DC. I also fixed a minor bug in utils.py that would occur when the collection only contained duplicates (see line 27 of utils.py).