search

gronlund / cvrdata

Extract data from danish CVR registry from Danish Business Authority
MIT License
7 stars 2 forks source link

some tables are empty #3

Open naarkhoo opened 5 years ago

naarkhoo commented 5 years ago

the code ended with the following message,

5289999 objects parsed and inserted into queue
5299999 objects parsed and inserted into queue
consumer timeout reached - retrying 
consumer timeout reached - retrying 
consumer timeout reached - retrying 
text/html
['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
<class 'str'>
Unknown mimetype, unable to deserialize: text/html
objects parsing done
Producer Done. Exiting...35608
Producer Time Used: 7112.439982175827
[INFO/Process-1] process shutting down
[INFO/Process-1] process exiting with exitcode 0
sentinel found - Thats it im out of here
Consumer Done. Exiting...35611 - time used 7112.4795389175415
[INFO/Process-4] process shutting down
[INFO/Process-4] process exiting with exitcode 0
sentinel found - Thats it im out of here
Consumer Done. Exiting...35609 - time used 7112.494404792786
sentinel found - Thats it im out of here
Consumer Done. Exiting...35610 - time used 7112.495819091797
[INFO/Process-2] process shutting down
[INFO/Process-2] process exiting with exitcode 0
[INFO/Process-3] process shutting down
[INFO/Process-3] process exiting with exitcode 0
[INFO/MainProcess] process shutting down

I wonder if this is the end, or it is due to some internal issues and probably I should re-run it ? thanks

gronlund commented 5 years ago

This means it is finished. This is some bug in elasticsearch that errors when my client sends a close signal. I think it should be done. Maybe i should try and handle this better at some point.

naarkhoo commented 5 years ago

Thanks - I look forward to hear about the issue on Regnskaber. Thank you again.

gronlund commented 5 years ago

I do not see any issue on my fork on regnskaber.

-Allan