fog / fog-google

Fog for Google Cloud Platform
MIT License
99 stars 146 forks source link

integration tests failing #592

Closed geemus closed 1 year ago

geemus commented 1 year ago

@Temikus - do you have any ideas on what might be happening with the integration tests? Seems to be failing when trying to bundle and reference the local stuff. It's not obvious to me that my recent merge impacted it, so I wasn't sure what else may have changed. You can see an example of the failure here: https://github.com/fog/fog-google/actions/runs/4767247086. Thanks!

Temikus commented 1 year ago

Bundler fails for some reason:

Using fog-google 1.19.0 from source at `.`
Error: The process '/opt/hostedtoolcache/Ruby/2.7.8/x64/bin/bundle' failed with exit code null
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Set up Ruby

I suspect the new Ruby 2.7.8 has some caveats. This is a local runner, so I should be able to debug a bit.

Temikus commented 1 year ago

Cannot repro on the runner either 🤔

Using fog-google 1.19.0 from source at `.`
Bundle complete! 14 Gemfile dependencies, 78 gems now installed.
Bundled gems are installed into `./vendor/bundle`
Temikus commented 1 year ago

Lemme try running from master to exclude any weirdness.

Temikus commented 1 year ago

The only difference with unit that passed is:

        env:
          # Needs to be set up for self-hosted runners, see:
          # https://github.com/ruby/setup-ruby#using-self-hosted-runners
          # Image used in runners: summerwind/actions-runner
          ImageOS: ubuntu20

Maybe actions-runner no longer needs it? Let's try it out I guess 🤔

Temikus commented 1 year ago

That didn't work either - kinda bizarre 🤔 I'll dig a bit more this evening.

Temikus commented 1 year ago

Now it's running fine in #594 🙃 I think there's something weird with our runners 🤔

geemus commented 1 year ago

Weird. Maybe just some kind of intermittent issue...

Temikus commented 1 year ago

I'm gonna try and add some debugging now and see how it goes on the main branch again.

Temikus commented 1 year ago

Pushed a new debug workflow in, I'll see what it spits out 🤔

Temikus commented 1 year ago

Ok, so something looks to be killing the runners:

Installing parser 3.2.2.1
/runner/_work/_temp/b9ca9963-76ef-47c5-b939-eab94c68eea3.sh: line 1:   354 Killed                  bundle install --verbose
Error: Process completed with exit code 137.

Looks like we're OOM'ing:

Screenshot 2023-04-25 at 6 28 55 pm
Memory cgroup out of memory: Killed process 2996984 (bundle) total-vm:777476kB, anon-rss:325672kB, file-rss:4792kB, shmem-rss:0kB, UID:1000 pgtables:860kB oom_score_adj:937
Temikus commented 1 year ago

Resized everything, will push a change soon as soon as the cluster rescales.

Temikus commented 1 year ago

Tests are looking much better - I will wait for current queue to finish and then restore caching and see if that was the reason for ruby/setup-ruby failing.

Temikus commented 1 year ago

Aight, tests are all green, trying with the ruby/setup-ruby 👍

geemus commented 1 year ago

@Temikus Awesome, thanks again for digging in and getting to the bottom of this.

Temikus commented 1 year ago

No worries, everything's looking good now ( except the usual Google API jitter :/ ) - I'll merge some fixes I made along the way soon 👍

geemus commented 1 year ago

Sounds good.