search

parca-dev / parca

Continuous profiling for analysis of CPU and memory usage, down to the line number and throughout time. Saving infrastructure cost, improving performance, and increasing reliability.
https://parca.dev/
Apache License 2.0
4.14k stars 213 forks source link

Report error "failed to insert profile: timestamp must not be zero" on v0.11.1 #1072

Open mangox opened 2 years ago

mangox commented 2 years ago

Error: level=error name=parca ts=2022-05-24T07:26:54.961974115Z caller=scrape.go:500 ... msg="WriteRaw failed for scraped profile" err="rpc error: code = Internal desc = failed to insert profile: timestamp must not be zero"

profiling config in parca.yml.

profiling_config:
      pprof_config:
        fgprof:
          enabled: true
          path: /debug/fgprof
          delta: true
knrt10 commented 2 years ago

I am not sure but I think I found the root cause. This line is adding seconds but subtracting 1 from it, due to which scraper is not picking it up correctly. As mentioned in parca.yaml it should be in ?seconds=<scrape-interval> format but instead it is in ?seconds=<scrape-interval> - 1.

I don't think this would solve the problem, doing this even removes /debug/pprof/profile WDYT cc @brancz?

brancz commented 2 years ago

Yeah unfortunately we realized that fgprof doesn't set several fields that we require to be set, I already opened several PRs to fix it: https://github.com/felixge/fgprof/pulls?q=is%3Apr+author%3Abrancz+

mangox commented 2 years ago

@brancz Cool. Looks those PRs could solve my issue. Hope they could be merged ASAP. I'll try to build it locally using your commits.

brancz commented 2 years ago

I tried reaching out to Felix via other channels as well, I know he’s super busy with $dayjob.