[Do not merge][graphbolt] s3-fifo-cache for DiskBasedFeature

dmlc / dgl

Python package built to ease deep learning on graph, on top of existing DL frameworks.

http://dgl.ai

Apache License 2.0

13.19k stars 2.99k forks source link

[Do not merge][graphbolt] s3-fifo-cache for DiskBasedFeature #7445

Open pyynb opened 1 month ago

pyynb commented 1 month ago

Description

Checklist

Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [$CATEGORY] (such as [NN], [Model], [Doc], [Feature]])
[ ] I've leverage the tools to beautify the python and c++ code.
[ ] The PR is complete and small, read the Google eng practice (CL equals to PR) to understand more about small PR. In DGL, we consider PRs with less than 200 lines of core code change are small (example, test and documentation could be exempted).
[ ] All changes have test coverage
[ ] Code is well-documented
[ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
[ ] Related issue is referred in this PR
[ ] If the PR is for a new model/paper, I've updated the example index here.

Changes

dgl-bot commented 1 month ago

To trigger regression tests:

@dgl-bot run [instance-type] [which tests] [compare-with-branch]; For example: @dgl-bot run g4dn.4xlarge all dmlc/master or @dgl-bot run c5.9xlarge kernel,api dmlc/master

mfbalin commented 1 month ago

Is the data required to run this code generated by some other code?

dgl-bot commented 1 month ago

Commit ID: 55d97f7cc140f95550bf36ba07a88fb6e1c3f80d

Build ID: 1

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot commented 1 month ago

Commit ID: 09dfc1fd42c6eea7dafe318f0683b343dec88825

Build ID: 2

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

pyynb commented 3 weeks ago

Is the data required to run this code generated by some other code?

Yes, during a training process, I recorded all the indexes and saved them in a txt file. The file is too large so i can't push it. You can try recording it yourself.

dgl-bot commented 3 weeks ago

Commit ID: b404ae34844d5cc7c75bc865a685ca6c4976d24d

Build ID: 3

Status: ❌ CI test failed in Stage [Distributed Torch CPU Unit test].

Report path: link

Full logs path: link

mfbalin commented 6 days ago

@pyynb Could you run the cache hit rate benchmark with the cache in #7492? It can be accessed via gb.impl.FeatureCache.

dgl-bot commented 3 days ago

Commit ID: 53107cfeba431ca74d2d539afe8339cb149bad65

Build ID: 4

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link