Remote build should retry launchpad queries

[lengau]

Bug Description

Sometimes launchpad queries getting a project, repo or recipe fails and just needs a backoff before trying again. We can probably retry 3 times before fully failing.

To Reproduce

Check charmcraft remote-build spread tests

part yaml

No response

Relevant log output

Total 18 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: Create a merge proposal for 'main' on Launchpad by visiting:
remote:       https://code.launchpad.net/~lengau/lengau-craft-remote-build/+git/charmcraft-charm-9bae065def77f3d680f53bbcb0154b9e/+ref/main/+register-merge
remote:
To https://git.launchpad.net/~lengau/lengau-craft-remote-build/+git/charmcraft-charm-9bae065def77f3d680f53bbcb0154b9e
 * [new branch]      master -> main
charmcraft internal error: ValueError("Could not find charm recipe 'charmcraft-charm-9bae065def77f3d680f53bbcb0154b9e' in project 'lengau-craft-remote-build' with owner 'lengau'")
Full execution log: '/root/.local/state/charmcraft/log/charmcraft-20240319-013323.238305.log'
Traceback (most recent call last):
  File "/snap/charmcraft/x1/bin/charmcraft", line 8, in <module>
    sys.exit(main())
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/application/main.py", line 141, in main
    return app.run()
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/application.py", line 476, in run
    return_code = dispatcher.run() or 0
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 487, in run
    return self._loaded_command.run(self._parsed_command_args)
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/commands/base.py", line 182, in run
    result = self._run(parsed_args, **kwargs) or result
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/application/commands/remote.py", line 125, in _run
    builds = builder.start_builds(project_dir)
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/services/remotebuild.py", line 91, in start_builds
    self._recipe = self._new_recipe(
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/services/remotebuild.py", line 75, in _new_recipe
    return self.lp.get_recipe(
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/launchpad/launchpad.py", line 161, in get_recipe
    return models.CharmRecipe.get(self, name, owner, project)
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/launchpad/models/recipe.py", line 398, in get
    raise ValueError(
ValueError: Could not find charm recipe 'charmcraft-charm-9bae065def77f3d680f53bbcb0154b9e' in project 'lengau-craft-remote-build' with owner 'lengau'

[syncronize-issues-to-jira[bot]]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-2626.

This message was autogenerated

[lengau]

Reopening this as we need to apply the some fix to repos and projects

[mr-cal]

I think we should also bump the sleep time from 3 seconds to 10-15 seconds. (Launchpad is moving slowly today and the total delay of 12 seconds (4 x 3 seconds) was not enough, but 30 seconds was enough)

[lengau]

Perhaps an exponential backoff and failing after 1 minute?