flexflow / FlexFlow

FlexFlow Serve: Low-Latency, High-Performance LLM Serving
https://flexflow.readthedocs.io
Apache License 2.0
1.59k stars 218 forks source link

Local Cost Estimator #1410

Closed reyna-abhyankar closed 2 weeks ago

reyna-abhyankar commented 3 weeks ago

Description of changes:

Add a local cost estimator that runs an operator's forward and backward pass on a single device and returns the elapsed time.

Related Issues:

Linked Issues:

Issues closed by this PR:


This change is Reviewable

lockshaw commented 2 weeks ago

lib/op-attrs/src/op-attrs/pcg_operator_attrs.cc line 18 at r3 (raw file):

Previously, reyna-abhyankar (Reyna Abhyankar) wrote…
How do I write this?
  [](BatchMatmulAttrs const &attrs) { ... },
  [](BatchNormAttrs const &attrs) { ... },
})```
lockshaw commented 2 weeks ago

lib/op-attrs/src/op-attrs/pcg_operator_attrs.cc line 18 at r3 (raw file):

Previously, lockshaw (Colin Unger) wrote…
```.visit(overload { [](BatchMatmulAttrs const &attrs) { ... }, [](BatchNormAttrs const &attrs) { ... }, })``` ```

Missing first line: .visit<ComputationGraphOpAttrs>(overload {

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 3.54610% with 136 lines in your changes missing coverage. Please review.

Project coverage is 39.28%. Comparing base (1a71d4c) to head (c00d75e).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## repo-refactor #1410 +/- ## ================================================= - Coverage 39.79% 39.28% -0.52% ================================================= Files 248 253 +5 Lines 8744 8859 +115 Branches 320 324 +4 ================================================= Hits 3480 3480 - Misses 5264 5379 +115 ``` | [Flag](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow) | `39.28% <3.54%> (-0.52%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow) | Coverage Δ | | |---|---|---| | [lib/kernels/include/kernels/allocation.h](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Fkernels%2Finclude%2Fkernels%2Fallocation.h&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL2tlcm5lbHMvaW5jbHVkZS9rZXJuZWxzL2FsbG9jYXRpb24uaA==) | `0.00% <ø> (ø)` | | | [...xecution/include/local-execution/local\_allocator.h](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Flocal-execution%2Finclude%2Flocal-execution%2Flocal_allocator.h&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL2xvY2FsLWV4ZWN1dGlvbi9pbmNsdWRlL2xvY2FsLWV4ZWN1dGlvbi9sb2NhbF9hbGxvY2F0b3IuaA==) | `0.00% <ø> (ø)` | | | [...n/include/local-execution/local\_training\_backing.h](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Flocal-execution%2Finclude%2Flocal-execution%2Flocal_training_backing.h&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL2xvY2FsLWV4ZWN1dGlvbi9pbmNsdWRlL2xvY2FsLWV4ZWN1dGlvbi9sb2NhbF90cmFpbmluZ19iYWNraW5nLmg=) | `0.00% <ø> (ø)` | | | [...cution/include/local-execution/op\_task\_signature.h](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Flocal-execution%2Finclude%2Flocal-execution%2Fop_task_signature.h&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL2xvY2FsLWV4ZWN1dGlvbi9pbmNsdWRlL2xvY2FsLWV4ZWN1dGlvbi9vcF90YXNrX3NpZ25hdHVyZS5o) | `0.00% <ø> (ø)` | | | [...cution/include/local-execution/tracked\_allocator.h](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Flocal-execution%2Finclude%2Flocal-execution%2Ftracked_allocator.h&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL2xvY2FsLWV4ZWN1dGlvbi9pbmNsdWRlL2xvY2FsLWV4ZWN1dGlvbi90cmFja2VkX2FsbG9jYXRvci5o) | `0.00% <ø> (ø)` | | | [lib/pcg/include/pcg/computation\_graph\_builder.h](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Fpcg%2Finclude%2Fpcg%2Fcomputation_graph_builder.h&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL3BjZy9pbmNsdWRlL3BjZy9jb21wdXRhdGlvbl9ncmFwaF9idWlsZGVyLmg=) | `100.00% <ø> (ø)` | | | [lib/pcg/src/pcg/computation\_graph.cc](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Fpcg%2Fsrc%2Fpcg%2Fcomputation_graph.cc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL3BjZy9zcmMvcGNnL2NvbXB1dGF0aW9uX2dyYXBoLmNj) | `15.38% <ø> (+2.88%)` | :arrow_up: | | [lib/pcg/src/pcg/computation\_graph\_builder.cc](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Fpcg%2Fsrc%2Fpcg%2Fcomputation_graph_builder.cc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL3BjZy9zcmMvcGNnL2NvbXB1dGF0aW9uX2dyYXBoX2J1aWxkZXIuY2M=) | `37.43% <100.00%> (ø)` | | | [lib/pcg/test/src/test\_computation\_graph\_builder.cc](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Fpcg%2Ftest%2Fsrc%2Ftest_computation_graph_builder.cc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL3BjZy90ZXN0L3NyYy90ZXN0X2NvbXB1dGF0aW9uX2dyYXBoX2J1aWxkZXIuY2M=) | `90.90% <100.00%> (ø)` | | | [...ion/include/local-execution/local\_cost\_estimator.h](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree&filepath=lib%2Flocal-execution%2Finclude%2Flocal-execution%2Flocal_cost_estimator.h&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow#diff-bGliL2xvY2FsLWV4ZWN1dGlvbi9pbmNsdWRlL2xvY2FsLWV4ZWN1dGlvbi9sb2NhbF9jb3N0X2VzdGltYXRvci5o) | `0.00% <0.00%> (ø)` | | | ... and [7 more](https://app.codecov.io/gh/flexflow/FlexFlow/pull/1410?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flexflow) | |