search

gronlund / cvrdata

Extract data from danish CVR registry from Danish Business Authority
MIT License
7 stars 2 forks source link

Producer timeout failed #2

Closed naarkhoo closed 5 years ago

naarkhoo commented 5 years ago

I wonder if it is normal to go through 100 interactions ? isn\t there any other work around ?!

for repeat in range(100):
                        try:
                            queue.put((dict_type, dat), timeout=60)
                            break
                        except Exception as e:
                            print('Producer timeout failed - retrying', repeat, e, dict_type, dat)
gronlund commented 5 years ago

this was a hack in reality it is not suppose to fail to insert. That should only happen if queue gets full but the consumers should be able to keep up with the producer. So a simple fail safe was made at some point. It may not be needed.