search

yugabyte / yb-voyager

Data migration Engine for YugabyteDB database
38 stars 10 forks source link

[Voyager] export data status panics #1866

Open jlisthood opened 1 week ago

jlisthood commented 1 week ago

Jira Link: DB-13971

Description

Context:

Issue: While running yb-voyager data export, I cannot check the status following the guide because I hit a panic, see below.

Logs:

[ssm-user@ip-10-71-251-29 bin]$ sudo -u ec2-user yb-voyager export data status --export-dir /home/ec2-user/migration
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/yugabyte/yb-voyager/yb-voyager/cmd.runExportDataStatusCmd()
        /home/ec2-user/rpmbuild/BUILD/yb-voyager-yb-voyager-v1.8.5-rc1/yb-voyager/cmd/exportDataStatusCommand.go:186 +0x905
github.com/yugabyte/yb-voyager/yb-voyager/cmd.init.func11(0x334ed80?, {0x1e7b330?, 0x4?, 0x1e7b1a0?})
        /home/ec2-user/rpmbuild/BUILD/yb-voyager-yb-voyager-v1.8.5-rc1/yb-voyager/cmd/exportDataStatusCommand.go:69 +0x25b
github.com/spf13/cobra.(*Command).execute(0x334ed80, {0xc000b0e120, 0x2, 0x2})
        /home/ec2-user/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x3351e60)
        /home/ec2-user/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
        /home/ec2-user/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/yugabyte/yb-voyager/yb-voyager/cmd.Execute()
        /home/ec2-user/rpmbuild/BUILD/yb-voyager-yb-voyager-v1.8.5-rc1/yb-voyager/cmd/root.go:208 +0x1a
main.main()
        /home/ec2-user/rpmbuild/BUILD/yb-voyager-yb-voyager-v1.8.5-rc1/yb-voyager/main.go:42 +0x45
[ssm-user@ip-10-71-251-29 bin]$

Note: the data export is running correctly:

calculating approx num of rows to export for each table...
Initiating data export.
Data export started.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

shubham-yb commented 6 days ago

Hi @jlisthood , I was trying out this flow and was able to repro this error on my end.

The entire flow goes like this

Running export data status before export data command is run:

yb-voyager export data status -e export-dir/
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xca88f7]

goroutine 1 [running]:
github.com/yugabyte/yb-voyager/yb-voyager/src/metadb.(*MetaDB).GetJsonObject(0x0, 0x0, {0x1e787ce, 0x10}, {0x1b40de0, 0xc0009a7900})
    /home/ubuntu/yb-voyager/yb-voyager/src/metadb/metadataDB.go:289 +0x157
github.com/yugabyte/yb-voyager/yb-voyager/src/metadb.(*MetaDB).GetMigrationStatusRecord(0x0)
    /home/ubuntu/yb-voyager/yb-voyager/src/metadb/migrationStatus.go:66 +0x4c
github.com/yugabyte/yb-voyager/yb-voyager/cmd.checkStreamingMode()
    /home/ubuntu/yb-voyager/yb-voyager/cmd/common.go:626 +0x1e
github.com/yugabyte/yb-voyager/yb-voyager/cmd.init.func11(0x3136da0?, {0x1e65e34?, 0x4?, 0x1e65ca4?})
    /home/ubuntu/yb-voyager/yb-voyager/cmd/exportDataStatusCommand.go:48 +0x25
github.com/spf13/cobra.(*Command).execute(0x3136da0, {0xc0008959a0, 0x2, 0x2})
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x3139e80)
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/yugabyte/yb-voyager/yb-voyager/cmd.Execute()
    /home/ubuntu/yb-voyager/yb-voyager/cmd/root.go:208 +0x1a
main.main()
    /home/ubuntu/yb-voyager/yb-voyager/main.go:42 +0x45

Export data command is run:

yb-voyager export data status -e export-dir/
Export Data Status for SourceDB

Export is said as started. We face this for a brief moment when the internal tool is starting up:

yb-voyager export data status -e export-dir/
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/yugabyte/yb-voyager/yb-voyager/cmd.runExportDataStatusCmd()
    /home/ubuntu/yb-voyager/yb-voyager/cmd/exportDataStatusCommand.go:186 +0x905
github.com/yugabyte/yb-voyager/yb-voyager/cmd.init.func11(0x3136da0?, {0x1e65e34?, 0x4?, 0x1e65ca4?})
    /home/ubuntu/yb-voyager/yb-voyager/cmd/exportDataStatusCommand.go:69 +0x25b
github.com/spf13/cobra.(*Command).execute(0x3136da0, {0xc0008a9a40, 0x2, 0x2})
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x3139e80)
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/yugabyte/yb-voyager/yb-voyager/cmd.Execute()
    /home/ubuntu/yb-voyager/yb-voyager/cmd/root.go:208 +0x1a
main.main()
    /home/ubuntu/yb-voyager/yb-voyager/main.go:42 +0x45

The above panic switches to:

yb-voyager export data status -e export-dir/
Export Data Status for SourceDB

TABLE       STATUS      EXPORTED ROWS
CATEGORIES  NOT-STARTED 0            
CUSTOMERS   NOT-STARTED 0            
CUSTOMERS1  NOT-STARTED 0            
CUSTOMERS2  NOT-STARTED 0            
CUSTOMERS3  NOT-STARTED 0            
CUST_HIST   NOT-STARTED 0            
INVENTORY   NOT-STARTED 0            
ORDERLINES  NOT-STARTED 0            
ORDERS      NOT-STARTED 0            
PRODUCTS    NOT-STARTED 0

Once we start seeing the progress bars:

yb-voyager export data status -e export-dir/
Export Data Status for SourceDB

TABLE       STATUS      EXPORTED ROWS
CUST_HIST   NOT-STARTED 0            
INVENTORY   NOT-STARTED 0            
ORDERLINES  NOT-STARTED 0            
ORDERS      NOT-STARTED 0            
PRODUCTS    NOT-STARTED 0            
CUSTOMERS   EXPORTING   210000       
CUSTOMERS1  EXPORTING   190000       
CUSTOMERS2  EXPORTING   210000       
CUSTOMERS3  EXPORTING   210000       
CATEGORIES  DONE        16

Could you confirm if you're observing the same behavior? Does the panic state persist, or does it resolve shortly after the export fully starts, given that these steps are executed in quick succession? We'll investigate both stack traces and work on addressing the panics. Also what's the source database type in your case? The above example is with Oracle as the source.

hbhanawat commented 5 days ago

Target release for the Voyager fix: 1.8.6 (Tentative date: 26th November).

jlisthood commented 4 days ago

Could you confirm if you're observing the same behavior? Does the panic state persist, or does it resolve shortly after the export fully starts, given that these steps are executed in quick succession? We'll investigate both stack traces and work on addressing the panics. Also what's the source database type in your case? The above example is with Oracle as the source

The panic state persisted while the export was making progress for many minutes. I left it be and after the export was fully done, I could run the command without panic

Also what's the source database type in your case? The above example is with Oracle as the source. Postgres v13 in this case

priyanshi-yb commented 4 days ago

Hi @jlisthood , As mentioned above we are already fixing the issue and will be released by EOM. Meanwhile, if you face this issue again, could you please share the contents of this file export-dir/metainfo/export_snapshot_status.json when you face the panic? It will help us see if there is anything else that is an issue. Thanks!