Why onnxruntime extracts only 483MB json file?

[Apisteftos]

Describe the issue

I am doing inference with my personal PC with batch_size 1 and profiling file reaches almost 483MB. Has any idea why is that happening? How can I increase the size of the profiled json file?

To reproduce

Profiling with DenseNet121 from onnx zoo on GPU. The exctracted json file reaches 483 and the stops to profile. An error occures:

2023-08-05 00:45:55.844032214 [E:onnxruntime:, profiler.cc:93 EndTimeAndRecordEvent] Maximum number of events reached, could not record profile event.

• How to prevent this event?
• How to increase the size of the output file?

I tried the same on a server and it reaches 2.2GB why this difference? I suppose it has with the SSD formating type?

Urgency

No response

Platform

Linux

OS Version

Ubuntu 22.04

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.15.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 11.6

[pranavsharma]

It has nothing to do with the size of the json file. It's the # of events. See https://github.com/microsoft/onnxruntime/blob/cf19c3697d126982acf45c040729269e53b5d9f4/onnxruntime/core/common/profiler.cc#L10

[Apisteftos]

Why in server is 2.2GB ? The number of the events is the same on my PC and server. I suppose it has with the inference time?

#else
profiling::Profiler::~Profiler() {}
#endif

::onnxruntime::TimePoint profiling::Profiler::Start() {
  ORT_ENFORCE(enabled_);
  auto start_time = std::chrono::high_resolution_clock::now();
  auto ts = TimeDiffMicroSeconds(profiling_start_time_, start_time);
  for (const auto& ep_profiler : ep_profilers_) {
    ep_profiler->Start(ts);
  }
  return start_time;
}

How to change the settings if there are provided?

[yufenglee]

Do you really need so many events? If you profile to investigate perf issue, you may try reducing the repeat count.