manifest file not taken into account by bootstrap command

[baycarbone]

Versions:

juju                3.4.4                    27554  3.4/stable       canonical✓       -
juju-db             4.4.24                   178    4.4/stable       juju-qa          -
maas                3.4.2-14353-g.5a5221d57  35359  3.4/stable       canonical✓       held
maas-anvil          0.1                      23     latest/edge      canonical✓       -

When running the bootstrap command and specifying a manifest file (e.g. to change some charm config options), it looks like the manifest file is ignored.

Manifest file:

# Generated Anvil Deployment Manifest

deployment:
  bootstrap:
    # Management networks shared by hosts (CIDRs, separated by comma)
    management_cidr: 172.31.16.0/20
software:
  # juju:
  #   bootstrap_args: []
  #   scale_args: []
  # charms:
  #   maas-region:
  #     channel: latest/edge
  #     revision: null
  #     config: null
  #   maas-agent:
  #     channel: latest/edge
  #     revision: null
  #     config: null
  #   haproxy:
  #     channel: latest/stable
  #     revision: null
  #     config: null
  #   postgresql:
  #     channel: 14/candidate
  #     revision: null
  #     config: null
    pgbouncer:
      channel: 1/candidate
      revision: null
      config:
        pool_mode: session
  # terraform:
  #   maas-region-plan:
  #     source: /snap/maas-anvil/23/etc/deploy-maas-region
  #   maas-agent-plan:
  #     source: /snap/maas-anvil/23/etc/deploy-maas-agent
  #   haproxy-plan:
  #     source: /snap/maas-anvil/23/etc/deploy-haproxy
  #   postgresql-plan:
  #     source: /snap/maas-anvil/23/etc/deploy-postgresql

Command: maas-anvil -v cluster bootstrap --manifest /home/ubuntu/.config/anvil/config.yaml --role database --role region --role agent --role haproxy

Log output: (full log: https://pastebin.canonical.com/p/K4t3TYQjgS/)

[12:08:24] DEBUG    Logging to '/home/ubuntu/snap/maas-anvil/common/logs/anvil-20240704-120824.253556.log'                                                                  log.py:84
           DEBUG    No juju account found                                                                                                                            deployment.py:70
                    Traceback (most recent call last):                                                                                                                               
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/sunbeam/jobs/juju.py", line 167, in load                                                                
                        with data_file.open() as file:                                                                                                                               
                      File "/usr/lib/python3.10/pathlib.py", line 1119, in open                                                                                                      
                        return self._accessor.open(self, mode, buffering, encoding, errors,                                                                                          
                    FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/snap/maas-anvil/23/account.yaml'                                                           

                    The above exception was the direct cause of the following exception:                                                                                             

                    Traceback (most recent call last):                                                                                                                               
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/sunbeam/provider/local/deployment.py", line 66, in _load_juju_account                                   
                        juju_account = JujuAccount.load(snaphelpers.Snap().paths.user_data)                                                                                          
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/sunbeam/jobs/juju.py", line 170, in load                                                                
                        raise JujuAccountNotFound(                                                                                                                                   
                    sunbeam.jobs.juju.JujuAccountNotFound: Juju user account not found, is node part of sunbeam cluster yet?                                                         
                    /home/ubuntu/snap/maas-anvil/23/account.yaml                                                                                                                     
           DEBUG    [get] http+unix://%2Fvar%2Fsnap%2Fmaas-anvil%2Fcommon%2Fstate%2Fcontrol.socket/1.0/config/JujuController, args={'allow_redirects': True}           service.py:120
           DEBUG    http://localhost:None "GET /1.0/config/JujuController HTTP/1.1" 503 131                                                                     connectionpool.py:474
           DEBUG    Response(<Response [503]>) = {"type":"error","status":"","status_code":0,"operation":"","error_code":503,"error":"Daemon not yet                   service.py:122
                    initialized","metadata":null}                                                                                                                                    

           DEBUG    Clusterd service unavailable                                                                                                                     deployment.py:80
                    Traceback (most recent call last):                                                                                                                               
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/sunbeam/clusterd/service.py", line 133, in _request                                                     
                        response.raise_for_status()                                                                                                                                  
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status                                                     
                        raise HTTPError(http_error_msg, response=self)                                                                                                               
                    requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url:                                                                                    
                    http+unix://%2Fvar%2Fsnap%2Fmaas-anvil%2Fcommon%2Fstate%2Fcontrol.socket/1.0/config/JujuController                                                               

                    During handling of the above exception, another exception occurred:                                                                                              

                    Traceback (most recent call last):                                                                                                                               
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/sunbeam/provider/local/deployment.py", line 75, in _load_juju_controller                                
                        return JujuController.load(self.get_client())                                                                                                                
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/sunbeam/jobs/juju.py", line 193, in load                                                                
                        controller = client.cluster.get_config(JUJU_CONTROLLER_KEY)                                                                                                  
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/sunbeam/clusterd/cluster.py", line 173, in get_config                                                   
                        return self._get(f"/1.0/config/{key}").get("metadata")                                                                                                       
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/sunbeam/clusterd/service.py", line 185, in _get                                                         
                        return self._request("get", path, **kwargs)                                                                                                                  
                      File "/snap/maas-anvil/23/lib/python3.10/site-packages/sunbeam/clusterd/service.py", line 158, in _request                                                     
                        raise ClusterServiceUnavailableException(                                                                                                                    
                    sunbeam.clusterd.service.ClusterServiceUnavailableException: Sunbeam Cluster not initialized                                                                     
           DEBUG    Plugin classes: {}                                                                                                                                   plugin.py:82
           DEBUG    Calling __post__init__                                                                                                                            manifest.py:102
           DEBUG    Plugin classes: {}                                                                                                                                   plugin.py:82
           DEBUG    Plugin classes: {}                                                                                                                                   plugin.py:82
           DEBUG    Plugin classes: {}                                                                                                                                   plugin.py:82
           DEBUG    Manifest used for deployment - preseed: {'bootstrap': {'management_cidr': '172.31.16.0/20'}}                                                      commands.py:188
           DEBUG    Manifest used for deployment - software: SoftwareConfig(juju=JujuManifest(bootstrap_args=[], scale_args=[]), charms={'maas-region':               commands.py:191
                    CharmsManifest(channel='latest/edge', revision=None, config=None), 'maas-agent': CharmsManifest(channel='latest/edge', revision=None,                            
                    config=None), 'haproxy': CharmsManifest(channel='latest/stable', revision=None, config=None), 'postgresql':                                                      
                    CharmsManifest(channel='14/candidate', revision=None, config=None), 'pgbouncer': CharmsManifest(channel='1/candidate', revision=None,                            
                    config=None)}, terraform={'maas-region-plan': TerraformManifest(source=PosixPath('/snap/maas-anvil/23/etc/deploy-maas-region')),                                 
                    'maas-agent-plan': TerraformManifest(source=PosixPath('/snap/maas-anvil/23/etc/deploy-maas-agent')), 'haproxy-plan':                                             
                    TerraformManifest(source=PosixPath('/snap/maas-anvil/23/etc/deploy-haproxy')), 'postgresql-plan':                                                                
                    TerraformManifest(source=PosixPath('/snap/maas-anvil/23/etc/deploy-postgresql'))})                                                                               
           DEBUG    Bootstrap node: roles DATABASE,REGION,AGENT,HAPROXY                                                                                               commands.py:208
           DEBUG    Starting pre-flight check Check for system requirements                                                                                             common.py:241
           DEBUG    Starting pre-flight check Check for juju snap                                                                                                       common.py:241
           DEBUG    Starting pre-flight check Check for ssh-keys interface                                                                                              common.py:241
           DEBUG    Starting pre-flight check Check for snap_daemon group membership                                                                                    common.py:241
           DEBUG    Starting pre-flight check Check for .local/share directory                                                                                          common.py:241
           DEBUG    Starting step 'Login to Juju controller'                                                                                                            common.py:260
           DEBUG    Local account not found, most likely not bootstrapped / joined                                                                                       juju.py:1328
           DEBUG    Skipping step Login to Juju controller                                                                                                              common.py:270
           DEBUG    Starting step 'Bootstrap Cluster'                                                                                                                   common.py:260
           DEBUG    [get] http+unix://%2Fvar%2Fsnap%2Fmaas-anvil%2Fcommon%2Fstate%2Fcontrol.socket/cluster/1.0/cluster, args={'allow_redirects': True}                 service.py:120
           DEBUG    http://localhost:None "GET /cluster/1.0/cluster HTTP/1.1" 503 131                                                                           connectionpool.py:474
           DEBUG    Response(<Response [503]>) = {"type":"error","status":"","status_code":0,"operation":"","error_code":503,"error":"Daemon not yet                   service.py:122
                    initialized","metadata":null}                                                                                                                                    

           DEBUG    Sunbeam Cluster not initialized                                                                                                                    clusterd.py:89
           DEBUG    Running step Bootstrap Cluster                                                                                                                      common.py:276
           DEBUG    [post] http+unix://%2Fvar%2Fsnap%2Fmaas-anvil%2Fcommon%2Fstate%2Fcontrol.socket/cluster/control, args={'data': '{"bootstrap": true, "address":     service.py:120
                    "172.31.19.209:7000", "name": "ip-172-31-19-209.eu-central-1.compute.internal"}', 'json': None}                                                                  
⠋ Bootstrapping Sunbeam cluster ...            DEBUG    http://localhost:None "POST /cluster/control HTTP/1.1" 200 108                                                                              connectionpool.py:474
           DEBUG    Response(<Response [200]>) = {"type":"sync","status":"Success","status_code":200,"operation":"","error_code":0,"error":"","metadata":{}}           service.py:122

           DEBUG    [post] http+unix://%2Fvar%2Fsnap%2Fmaas-anvil%2Fcommon%2Fstate%2Fcontrol.socket/1.0/nodes, args={'data': '{"name":                                 service.py:120
                    "ip-172-31-19-209.eu-central-1.compute.internal", "role": ["database", "region", "agent", "haproxy"], "machineid": 0, "systemid": ""}', 'json':                  
                    None}                                                                                                                                                            
           DEBUG    http://localhost:None "POST /1.0/nodes HTTP/1.1" 200 108                                                                                    connectionpool.py:474
           DEBUG    Response(<Response [200]>) = {"type":"sync","status":"Success","status_code":200,"operation":"","error_code":0,"error":"","metadata":{}}           service.py:122

           DEBUG    Finished running step 'Bootstrap Cluster'. Result: ResultType.COMPLETED                                                                             common.py:279
           DEBUG    Starting step 'Write Manifest to database'                                                                                                          common.py:260
           DEBUG    [get] http+unix://%2Fvar%2Fsnap%2Fmaas-anvil%2Fcommon%2Fstate%2Fcontrol.socket/1.0/manifests/latest, args={'allow_redirects': True}                service.py:120
           DEBUG    http://localhost:None "GET /1.0/manifests/latest HTTP/1.1" 404 127                                                                          connectionpool.py:474
           DEBUG    Response(<Response [404]>) = {"type":"error","status":"","status_code":0,"operation":"","error_code":404,"error":"ManifestItem not                 service.py:122
                    found","metadata":null}                                                                                                                                          

           DEBUG    Running step Write Manifest to database                                                                                                             common.py:276
           DEBUG    [post] http+unix://%2Fvar%2Fsnap%2Fmaas-anvil%2Fcommon%2Fstate%2Fcontrol.socket/1.0/manifests, args={'data': '{"manifestid":                       service.py:120
                    "584c551c18f7d166ab795cb3cad9360f", "data": "deployment:\\n  bootstrap:\\n    management_cidr: 172.31.16.0/20\\nsoftware:\\n  pgbouncer:\\n                      
                    channel: 1/candidate\\n    config:\\n      pool_mode: session\\n    revision: null\\n"}', 'json': None}                                                          
           DEBUG    http://localhost:None "POST /1.0/manifests HTTP/1.1" 200 108                                                                                connectionpool.py:474
           DEBUG    Response(<Response [200]>) = {"type":"sync","status":"Success","status_code":200,"operation":"","error_code":0,"error":"","metadata":{}}           service.py:122

           DEBUG    Finished running step 'Write Manifest to database'. Result: ResultType.COMPLETED                                                                    common.py:279
           DEBUG    Starting step 'Add Cloud'                                                                                                                           common.py:260
           DEBUG    Running command /snap/maas-anvil/23/juju/bin/juju clouds --client --format json                                                                        juju.py:91
⠇ Adding cloud to Juju client ... [12:08:38] DEBUG    Command finished. stdout={"localhost":{"defined":"built-in","type":"lxd","description":"LXD Container Hypervisor","auth-types":["certificate"],"regions":{"localhost":{}}}}                      juju.py:93
                    , stderr=Since Juju 3 is being run for the first time, it has downloaded the latest public cloud information.                                                                                              
                    Only clouds with registered credentials are shown.                                                                                                                                                         
                    There are more clouds, use --all to see them.       
...

[skatsaounis]

Hi @baycarbone ,

The mainfest.yaml you have provided has a commented out line which shouldn't be commented out: # charms:.

I tried with the following manifest.yaml:

deployment:
  bootstrap:
    management_cidr: 10.20.0.0/24
  haproxy:
    virtual_ip: 10.20.0.80
  postgres:
    max_connections: 110
software:
  charms:
    postgresql:
      config:
        experimental_max_connections: 120
    pgbouncer:
      channel: 1/stable
      config:
        pool_mode: transaction

Which is overriding many things, including the final max_connections value that will be used for postgresql. After a successful bootstrap, I was able to observer that instead of default pool_mode: session and experimental_max_connections: 110 that Anvil typically sets, the overridden values where chosen:

ubuntu@maas-1:~$ juju config pgbouncer pool_mode
transaction
ubuntu@maas-1:~$ juju config postgresql experimental_max_connections
120
ubuntu@maas-1:~$ vim .config/anvil/manifest.yaml

Full logs are available here: https://paste.ubuntu.com/p/tCWyqN4hNX/

I would suggest to try again with a proper manifest.yaml, if you still have the need for explicitly overriding Anvil values.

Thank you for your time to report this issue.