Feature/gc ddf - Githubissues

ddf-project / DDF

Distributed DataFrame: Productivity = Power x Simplicity For Scientists & Engineers, on any Data Engine

http://ddf.io

Apache License 2.0

167 stars 42 forks source link

Feature/gc ddf #351

Closed Huandao0812 closed 8 years ago

Huandao0812 commented 8 years ago

Description and related tickets, documents

DDFCache is backed by a guava's Loading cache, https://github.com/google/guava/wiki/CachesExplained The eviction policy is LRU For DDF cache we use both size-based eviction and time eviction:

cache's capacity is set by MAX_NUMBER_OF_DDFS_IN_CACHE, the cache will evict DDF that haven't been used recently.
Evict DDF after DDF_TIME_TO_IDLE_SECONDS seconds have passed since it last accessed.
Restore DDF by using DDFManager.restore function if there's request for DDF after it was evicted

Reviewers: @zkidkid @nhanitvn @phvu @lebinh @ubolonton

PR Progress

Make sure all checkboxes below are checked before merged

[x] Branch is in format prefix/description (see this)
[x] Merge check has no conflicts. PR checks passed.
[x] Code review is done.

phvu commented 8 years ago

lgtm

nhanitvn commented 8 years ago

lgtm

ubolonton commented 8 years ago

This creates the issue of important not-used-recently DDFs being prioritized for removal over recent temporary DDFs. While it is probably ok for the use case we are fixing, it will likely cause spikes of performance degradation for other use cases.
Can we make this a handler so that PE can customize it? For example it can use a different caching strategy that takes a "cache"/"pin" flag into account, which would be specific to PE's use cases, not DDF core.

Huandao0812 commented 8 years ago

@ubolonton we could extends and override the class, one simple solution is adding a cache flag to signal don't remove this DDF

ducleminh commented 8 years ago

@ubolonton @Huandao0812 or don't cache the temporary DDFs if we can work out whether they are temporary.

nhanitvn commented 8 years ago

retest this please

hai-adatao commented 8 years ago

Hold on, don't merge this one yet

Huandao0812 commented 8 years ago

I change the DDFCache to keep DDF with name on a separate cache, let's go with this first, a separate flag to signal long live versus ephemeral can be add later. Please review @nhanitvn @phvu @ubolonton @zkidkid @lebinh

nhanitvn commented 8 years ago

That would be a better approach. Anyhow, we have no-named DDFs and named DDFs.