[BUG] Error watching metadata: invalid character '<' looking for beginning of value.

[Sachinbisht27]

Describe the bug While performing the regular migrations for the WhatsApp webhook service, the AirByte instance encountered a few issues. As a result, the errors were logged in the logging and monitoring dashboard. We need to diagnose why these errors are occurring.

To Reproduce Steps to reproduce the behavior:

1. Go to the logging and monitoring dashboard )and see the error log on the AirByte instance logs.

Expected behavior

• The service to migrate the data should migrate data without any errors.

Screenshots

Environment (please complete the following information):

• Environment - Production

Additional context JSON log:

{
  "insertId": "vo71hof47vek8",
  "jsonPayload": {
    "message": "Error watching metadata: invalid character '<' looking for beginning of value",
    "omitempty": null,
    "localTimestamp": "2024-10-30T03:56:18.3910Z"
  },
  "resource": {
    "type": "gce_instance",
    "labels": {
      "zone": "asia-south1-a",
      "project_id": "cryptic-gate-211900",
      "instance_id": "258947889300566770"
    }
  },
  "timestamp": "2024-10-30T03:56:18.402170158Z",
  "severity": "ERROR",
  "labels": {
    "instance_name": "instance-1"
  },
  "logName": "projects/cryptic-gate-211900/logs/GCEGuestAgent",
  "sourceLocation": {
    "file": "metadata.go",
    "line": "74",
    "function": "github.com/GoogleCloudPlatform/guest-agent/google_guest_agent/events/metadata.(*Watcher).Run"
  },
  "receiveTimestamp": "2024-10-30T03:56:19.531959520Z"
}

Acceptance Criteria

• [x] The root cause of the error should be reported and shared with the team.

Documentation Add whatever documentation will be required here.

[Sachinbisht27]

Diagnosis Report

Problem

When migrating data for the WhatsApp webhook service, errors appeared in the logs. These errors stopped the migration from completing successfully, affecting the data used for WhatsApp analytics.

Example Error Log:

"Error watching metadata: invalid character '<' looking for beginning of value"

Cause of the Problem

1. Some custom fields in the database had special characters in their names.
2. These special characters could not be migrated to the target data source (BigQuery).
3. This mismatch caused the migration process to fail.

Impact

• The WhatsApp analytics service isn't getting the data it needs because of this migration error.
• Data used for insights and reporting is incomplete.

What We Fixed

1. Replaced special characters in the custom field names with underscores (_).
2. Removed database entries meant only for testing.

Result

After making these fixes, the data migration worked correctly, and the analytics service started receiving data as expected.