search

rdp / google_hash

wrapper for google's hash functions, for ruby
BSD 3-Clause "New" or "Revised" License
92 stars 16 forks source link

Google hash error killed my process #17

Closed kostyasys closed 13 years ago

kostyasys commented 13 years ago

Today my process, where i use google_hash died with the following error.

er (localhost:4000): ./local_installed/include/google/sparsehash/densehashtable.h:835: std::pair<google::dense_hashtable_iterator<V, K, HF, ExK, SetK, EqK, A>, bool> google::dense_hashtable<Value, Key, HashFcn, ExtractKey, SetKey, EqualKey, Alloc>::insert_noresize(const Value&) [with Value = std::pair<const int, int>, Key = int, HashFcn = std::tr1::hash, ExtractKey = google::dense_hash_map<int, int, std::tr1::hash, std::equal_to, std::allocator >::SelectKey, SetKey = google::dense_hash_map<int, int, std::tr1::hash, std::equal_to, std::allocator >::SetKey, EqualKey = std::equal_to, Alloc = std::allocator]: Assertion `!use_empty || !equals(get_key(obj), get_key(emptyval))' failed.

I suspect it happend because of large hashes. Process utilized about 2Gb ram. it stores all data in GoogleHashDenseIntToInt hash tables.

Can you explain why coud this happen and suggest possible solutions. May be i should shard my data between set of hashes and etc.

I use your library for few month in different cases, but i have never seen this error before.

rdp commented 13 years ago

google hash sets up a default "empty" key that should never be used...perhaps it seems that somehow here it is being used, despite my small effort to make that difficult...I might be able to fix it...do you know exactly which key is causing the conflict?

kostyasys commented 13 years ago

i can't reproduce this error in my testing environment. I will try to log some data in production, but i might take few days to catch it.

rdp commented 13 years ago

are you on 64-bit? could you paste your contents of dense_int_to_int.cpp somewhere so I can see it? -r

kostyasys commented 13 years ago

yes 64.

https://gist.github.com/744649

rdp commented 13 years ago

wow I was able to reproduce what "may" have been the problem.

https://gist.github.com/745567

Which is fixed with 0.6.2 If you're on 64 bit, another option is to use the "LongToLong" or what not, if you need larger numbers. Let me know how it goes. Cheers! -r

kostyasys commented 13 years ago

Thanks a lot for your support. Now it seams to be ok.