cirruslabs / cirrus-ci-agent

Agent to execute Cirrus CI tasks
Mozilla Public License 2.0
13 stars 6 forks source link

Bug Report: rpc error: code = ResourceExhausted desc = grpc: received message larger than max #244

Closed barkingfoodog closed 2 years ago

barkingfoodog commented 2 years ago

Environment

$ cirrus --version
cirrus version 0.84.0-ae08bae
$ tart --version
0.19.0
$ system_profiler SPSoftwareDataType SPHardwareDataType
Software:

    System Software Overview:

      System Version: macOS 12.5.1 (21G83)
      Kernel Version: Darwin 21.6.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      User Name: ***********
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled
      Time since boot: 7 days 12:43

Hardware:

    Hardware Overview:

      Model Name: Mac Studio
      Model Identifier: Mac13,1
      Chip: Apple M1 Max
      Total Number of Cores: 10 (8 performance and 2 efficiency)
      Memory: 64 GB
      System Firmware Version: 7459.141.1
      OS Loader Version: 7459.141.1
      Serial Number (system): ***********
      Hardware UUID: ***********
      Provisioning UDID: ***********
      Activation Lock Status: Enabled

Steps to reproduce

Create a .cirrus.yml file with the following contents:

task:
   macos_instance:
     image: ghcr.io/cirruslabs/macos-monterey-base:latest
     no-graphics: true
   name: example
   example_script:
     - base64 /dev/urandom | head -c 8388608 > example.txt || true
     - cat example.txt

Run with simple logging:

cirrus run -o simple

Expected behavior

Random data is output and the process terminates normally.

Actual behavior

An error relating to the rpc message size occurs:

Error while receivieng logs: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4664590 vs. 4194304)

Notes

This is a contrived example to demonstrate an error encountered converging a salt project with verbose logging. Other projects with smaller output don't exhibit this issue.

Generating 8MB of random output seems to consistently reproduce this issue. Smaller values, like 4MB + 1 byte, don't generate the error.

This seems like an easy fix to adjust the grpc message size, but my go-fu isn't up to the task.

Helpful resources/projects with a similar error:

barkingfoodog commented 2 years ago

Thanks for the quick fix @edigaryev and @fkorotkov !