yaml parsing errors in CLI tool `metadata export`

[alecf]

Version Information

Server Version: 2.32.1 CLI Version (for CLI related issue): 2.32.1

Environment

Cloud

What is the current behaviour?

Starting with version 2.32.1, I can no longer export metadata, I get an error parsing:

FATA[0000] failed to export metadata: cannot export sources from metadata: error parsing metadata 
object: sources
file: databases.yaml
error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!map into []string 

What is the expected behaviour?

Metadata export should work

How to reproduce the issue?

My databases.yaml (in my projects' metadata/databases)

- name: imaginary-db
  kind: postgres
  configuration:
    connection_info:
      database_url:
        from_env: PG_DB_CONNECTION_URL
      isolation_level: read-committed
      pool_settings:
        connection_lifetime: 60
        total_max_connections: 10
      use_prepared_statements: false
  tables: "!include imaginary-db/tables/tables.yaml"
  functions: "!include imaginary-db/functions/functions.yaml"

Download the v2.32.1 binary:

$ ./cli-hasura-darwin-arm64 version && ./cli-hasura-darwin-arm64 --envfile ../.env metadata export --project hasura --log-level DEBUG
INFO hasura cli                                    version=v2.32.1
DEBU global config file exists, verifying contents 
DEBU global config is pre-set to &cli.GlobalConfig{UUID:"2c7618a3-e384-4bd2-84b6-dcef928f2edb", EnableTelemetry:true, ShowUpdateNotification:true, CLIEnvironment:"default"} 
DEBU global config: uuid: 2c7618a3-e384-4bd2-84b6-dcef928f2edb 
DEBU global config: enableTelemetry: true         
DEBU global config: showUpdateNotification: true  
DEBU global config: cliEnvironment: default       
DEBU ENV vars read from: /Users/alecf/prompt-js/prompt-watch/.env 
DEBU versions: cli: [v2.32.1] server: [v2.32.1-cloud.1] 
DEBU compatibility check: [true] compatible CLI and Server 
DEBU graphql engine endpoint: https://imaginary-dev.hasura.app 
DEBU graphql engine admin_secret: ‹×›             
DEBU server: uuid: 876e1d60-c29e-40c4-8151-ddb28ac2e0eb 
DEBU telemetry: beamed                             fields.time=624.621625ms isError=true
DEBU failed to export metadata: cannot export sources from metadata: error parsing metadata 
object: sources
file: databases.yaml
error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!map into []string  kind="other error" location="file: /workdir/cli/internal/metadataobject/sources/sources.go, line: 256" ops="[command: hasura metadata export.RunE commands.MetadataExportOptions.Run commands.metadataModeDirectoryHandler.Export projectmetadata.Handler.ExportMetadata sources.SourceConfig.Export]"
FATA[0000] failed to export metadata: cannot export sources from metadata: error parsing metadata 
object: sources
file: databases.yaml
error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!map into []string 

But if I revert to v2.32.0, the export works:

./cli-hasura-darwin-arm64 version && ./cli-hasura-darwin-arm64 --envfile ../.env metad
ata export --project hasura --log-level DEBUG
INFO hasura cli                                    version=v2.32.0
DEBU global config file exists, verifying contents 
DEBU global config is pre-set to &cli.GlobalConfig{UUID:"2c7618a3-e384-4bd2-84b6-dcef928f2edb", EnableTelemetry:true, ShowUpdateNotification:true, CLIEnvironment:"default"} 
DEBU global config: uuid: 2c7618a3-e384-4bd2-84b6-dcef928f2edb 
DEBU global config: enableTelemetry: true         
DEBU global config: showUpdateNotification: true  
DEBU global config: cliEnvironment: default       
DEBU ENV vars read from: /Users/alecf/prompt-js/prompt-watch/.env 
DEBU versions: cli: [v2.32.0] server: [v2.32.1-cloud.1] 
DEBU compatibility check: [true] compatible CLI and Server 
DEBU graphql engine endpoint: https://imaginary-dev.hasura.app 
DEBU graphql engine admin_secret: ‹×›             
DEBU server: uuid: 876e1d60-c29e-40c4-8151-ddb28ac2e0eb 
DEBU cli-ext: setting up using cli-ext embedded in cli binary 
DEBU output: {"success":true,"output_file_path":"/var/folders/w_/b7kb19452pz062ssq59rytnc0000gn/T/619559982.json"}  command="sdl to"
INFO Metadata exported                            
DEBU telemetry: beamed                             fields.time=621.546375ms isError=false

Also note that v2.33.0 and v2.33.1 both exhibit this problem, and v2.30 and v2.31 do not exhibit this problem. So this was introduced in v2.32.1 and has not been fixed.

Screenshots or Screencast

see above for logs

Please provide any traces or logs that could help here.

Any possible solutions/workarounds you're aware of?

Sticking with v2.32.0

Keywords

yaml, parsing, metadata, export, binary, cloud

[alecf]

Looks like the error was introduced here: https://github.com/hasura/graphql-engine/commit/cb8b6235bfa806dd6eb4e7b39341eb9b595f184e#diff-60b2310f2876d8a440a4498e7ebe319e545b07ab71f8a022a14f019da59bdc00R238

the problem seems to be that the code previously assumed the functionNode could only be a DocumentNode when encountering functions... and it is falling back to trying to make this a snowflake database (mine is postgres)

I have postgres functions defined in my functions.yaml

- "!include public_accept_organization_invite.yaml"
- "!include public_initialize_test_run.yaml"

So I think this basically manifests itself on non-snowflake dbs with at least one tracked function.

[alecf]

Tagging @manasag who seems to be the original author.. hoping there's an easy fix!

[lazakrisz]

Hi, I also seem to be having this exact same issue and I've also been able to narrow it down to the functions.yml file. I'm also using a postgres database with hasura cli running at the following version: hasura-cli@2.32.1 Server version is: 2.31.0

I have the following config: (hasura/metadata/databases/databases.yaml)

- name: default
  kind: postgres
  configuration:
    connection_info:
      database_url:
        from_env: HASURA_GRAPHQL_DATABASE_URL
      isolation_level: read-committed
      use_prepared_statements: false
  tables: "!include default/tables/tables.yaml"
  functions: "!include default/functions/functions.yaml"

If I run hasura md reload then the output is the following:

INFO Metadata reloaded
INFO Metadata is consistent

However when I run hasura md export I get the exact same output as @alecf,

FATA[0001] failed to export metadata: cannot export sources from metadata: error parsing metadata
object: sources
file: databases.yaml
error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!map into []string

Upon experimenting further I've found that removing the functions from the hasura/metadata/databases/databases.yaml file and also removing the tracked functions through migrations does solve the yaml parsing issue:

> hasura migrate status
1684403470148  xyz Present        Present                                                                                                                                                                        
# ...
1693742401715  add domain and function Present        Present      # <--- this is the migration which includes the function + tracking                                                                                                                                                                   
> hasura migrate apply --type down --version 1693742401715
INFO migrations applied
> hasura md export
FATA[0000] failed to export metadata: cannot export sources from metadata: error parsing metadata
object: sources
file: databases.yaml
error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!map into []string
> hasura md ic list
INFO metadata is consistent
> hasura md reload
INFO Metadata reloaded
WARN Metadata is inconsistent, use 'hasura metadata ic list' command to see the inconsistent objects
> hasura md ic drop
INFO all inconsistent objects removed from metadata
> hasura md export
INFO Metadata exported

Downgrading to cli version 2.32.0 does fix the issue!

[lazakrisz]

I also found the following topic on stackoverflow: https://stackoverflow.com/a/77031360/19350095

[Dex19]

I encountered the same issue after updating Hasura to version 2.33.0-ce and cli to 2.33.0. After the update, I started getting errors.

FATA[0000] failed to export metadata: cannot export sources from metadata: error parsing metadata
object: sources
file: databases.yaml
error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!map into []string 

Additionally, I'm unable to make any changes in the Hasura console. Adding new columns is not recorded in metadata nor as migrations. When I try to delete a column or drop a table, I get the following error:

Hasura console is not able to reach your Hasura GraphQL engine instance. 
Please ensure that your instance is running and the endpoint is configured correctly.

Downgrading hasura cli to version 2.32.0 fixed all issues, I didn't dowgrade hasura version.

[rikinsk]

Folks, thanks for reporting. We have a fix for this and will be releasing a patch release soon.

[rikinsk]

Folks, patches for v2.32 and v2.33 have been released with fixes for this. You will just need to update the CLI for the fix to work.

[nbouvrette]

I thought I was going crazy with this bug.. lost half my day 😢

I confirm that the patch is working after running hasura update-cli. Thanks @rikinsk