ben-manes / concurrentlinkedhashmap

A ConcurrentLinkedHashMap for Java
Apache License 2.0
470 stars 113 forks source link

change weight references from int to long to support map capacity based on byte size that is > 2GB #24

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago
Current Behavior:
integer weight when used for maps that have their capacity based on byte size 
mean that the map is limited to 2gb.

Desired Behavior:
change all references to weight from and int to a long

Complexity / Amount of Change Required:
took me about 5 minuites in eclipse to change all the references and method 
return types in all the required methods of the concurrentlinkedhashmap 
,weighters and weighther  classes.

Original issue reported on code.google.com by pariodeu...@googlemail.com on 20 Apr 2011 at 1:30

GoogleCodeExporter commented 9 years ago
The Collections framework supports only an int #size(). If the default 
Weighers#singleton() is used, then the map's capacity is limited to its maximum 
size. This means that there could be unexpected behavior if the capacity allows 
the hash-table to exceed its maximum size.

The other concern with this change is that it increases memory used per-entry 
(int->long). This may be wasteful for the common case usages.

The easiest solution, if acceptable, is to change the base. If the weight was 
in terms of kilobytes then map is limited to 1TB. This seems like a more 
reasonable solution to me. Would that work for your use-case?

Original comment by Ben.Manes@gmail.com on 20 Apr 2011 at 2:37

GoogleCodeExporter commented 9 years ago
just dividing the weight by 1024 for weights based on byte size would be an 
acceptable solution. 

However having a large map > 2GB with values with average sizes between 500 
bytes and 1500 bytes would mean that the eviction procedure when the map is 
full would loose some performance, as the minimum weight allowed would be 1 
(1kb) so 2 or more smaller values could be evicted instead of one large value 
if the weight is 1 on all of them.

Original comment by pariodeu...@googlemail.com on 20 Apr 2011 at 9:33

GoogleCodeExporter commented 9 years ago
Estimating a good sector size (256b -> 512gb) is my preference then. The small 
waste is offset by the cache's size, which would most certainly evict a very 
cold entry.

Original comment by Ben.Manes@gmail.com on 20 Apr 2011 at 10:11