search

ROCm / rocprofiler-compute

Advanced Profiling and Analytics for AMD Hardware
https://rocm.docs.amd.com/projects/omniperf/en/latest/
MIT License
135 stars 49 forks source link

Metric filtering does not work with baseline compare #300

Closed JoseSantosAMD closed 3 weeks ago

JoseSantosAMD commented 7 months ago

Describe the bug Metric filtering does not work with baseline compare

Development Environment:

To Reproduce Steps to reproduce the behavior:

  1. Run omniperf analyze -p tests/workloads/vcopy/MI100/ -p tests/workloads/vcopy/MI200 --block 11
  2. See error numpy.core._exceptions._UFuncNoLoopError: ufunc 'add' did not contain a loop with signature matching types (dtype('float64'), dtype('<U2')) -> None
  3. Also returns error when passing same path

Screenshots image image

Additional context Add any other context about the problem here.

coleramos425 commented 7 months ago

I couldn't reproduce the issue with the commands you attached. I suspect you may have incorrectly pulled - I'll schedule an appointment to review over the phone

skyreflectedinmirrors commented 7 months ago

I hit the same issue today on:

commit 62221383a701f13fac63bdbf54b8fc6fb87ebc5e (HEAD -> 2.x, origin/2.x)
Author: colramos-amd <colramos@amd.com>
Date:   Mon Mar 18 14:28:14 2024 -0500

    Fixing visualization for Compute Pipeline SOL. Need to consider 'Pct of Peak' rather than 'Avg'

    Signed-off-by: colramos-amd <colramos@amd.com>

will put details in our chat

coleramos425 commented 7 months ago

If I provide the same path twice, error is correctly caught, i.e.

$ omniperf analyze -p workloads/mix/MI200/ -p workloads/mix/MI200/

  ___                  _                  __ 
 / _ \ _ __ ___  _ __ (_)_ __   ___ _ __ / _|
| | | | '_ ` _ \| '_ \| | '_ \ / _ \ '__| |_ 
| |_| | | | | | | | | | | |_) |  __/ |  |  _|
 \___/|_| |_| |_|_| |_|_| .__/ \___|_|  |_|  
                        |_|                  

Analysis mode = cli
ERROR You cannot provide the same path twice.

If I provide two different workloads and block filter, data correctly loads, i.e.

$ omniperf analyze -p workloads/mix/MI200/ -p workloads/mix_k_filt/MI200/ -b 11

  ___                  _                  __ 
 / _ \ _ __ ___  _ __ (_)_ __   ___ _ __ / _|
| | | | '_ ` _ \| '_ \| | '_ \ / _ \ '__| |_ 
| |_| | | | | | | | | | | |_) |  __/ |  |  _|
 \___/|_| |_| |_|_| |_|_| .__/ \___|_|  |_|  
                        |_|                  

Analysis mode = cli
[analysis] deriving Omniperf metrics...
...
--------------------------------------------------------------------------------
11. Compute Units - Compute Pipeline
11.1 Speed-of-Light
╒═════════════╤═══════════════════╤═════════╤═══════════════════╤════════════╤════════╤═══════════╤═════════════════╤═══════════════╤════════════════╕
│ Metric_ID   │ Metric            │     Avg │ Avg               │   Abs Diff │ Unit   │      Peak │ Peak            │   Pct of Peak │ Pct of Peak    │
╞═════════════╪═══════════════════╪═════════╪═══════════════════╪════════════╪════════╪═══════════╪═════════════════╪═══════════════╪════════════════╡
│ 11.1.0      │ VALU FLOPs        │ 7497.89 │ 10053.16 (34.08%) │      11.29 │ Gflop  │  22630.40 │ 22630.4 (0.0%)  │         33.13 │ 44.42 (34.08%) │
├─────────────┼───────────────────┼─────────┼───────────────────┼────────────┼────────┼───────────┼─────────────────┼───────────────┼────────────────┤
│ 11.1.1      │ VALU IOPs         │ 2325.84 │ 1132.67 (-51.3%)  │      -5.27 │ Giop   │  22630.40 │ 22630.4 (0.0%)  │         10.28 │ 5.01 (-51.28%) │
├─────────────┼───────────────────┼─────────┼───────────────────┼────────────┼────────┼───────────┼─────────────────┼───────────────┼────────────────┤
│ 11.1.2      │ MFMA FLOPs (BF16) │    0.00 │ 0.0 (0.0%)        │       0.00 │ Gflop  │ 181043.20 │ 181043.2 (0.0%) │          0.00 │ 0.0 (0.0%)     │
├─────────────┼───────────────────┼─────────┼───────────────────┼────────────┼────────┼───────────┼─────────────────┼───────────────┼────────────────┤
│ 11.1.3      │ MFMA FLOPs (F16)  │    0.00 │ 0.0 (0.0%)        │       0.00 │ Gflop  │ 181043.20 │ 181043.2 (0.0%) │          0.00 │ 0.0 (0.0%)     │
├─────────────┼───────────────────┼─────────┼───────────────────┼────────────┼────────┼───────────┼─────────────────┼───────────────┼────────────────┤
│ 11.1.4      │ MFMA FLOPs (F32)  │    0.00 │ 0.0 (0.0%)        │       0.00 │ Gflop  │  45260.80 │ 45260.8 (0.0%)  │          0.00 │ 0.0 (0.0%)     │
├─────────────┼───────────────────┼─────────┼───────────────────┼────────────┼────────┼───────────┼─────────────────┼───────────────┼────────────────┤
│ 11.1.5      │ MFMA FLOPs (F64)  │    0.00 │ 0.0 (0.0%)        │       0.00 │ Gflop  │  45260.80 │ 45260.8 (0.0%)  │          0.00 │ 0.0 (0.0%)     │
├─────────────┼───────────────────┼─────────┼───────────────────┼────────────┼────────┼───────────┼─────────────────┼───────────────┼────────────────┤
│ 11.1.6      │ MFMA IOPs (INT8)  │    0.00 │ 0.0 (0.0%)        │       0.00 │ Giop   │ 181043.20 │ 181043.2 (0.0%) │          0.00 │ 0.0 (0.0%)     │
╘═════════════╧═══════════════════╧═════════╧═══════════════════╧════════════╧════════╧═══════════╧═════════════════╧═══════════════╧════════════════╛
11.2 Pipeline Stats
...

I may need some extra help understanding where this issue is occuring

ppanchad-amd commented 3 weeks ago

@JoseSantosAMD Issue is fixed. Closing ticket. Thanks!