message_id not populated in systemProperties when reading from eventhub

[jimtodd92]

Bug Report:

• Actual behavior: message_id is not seen in systemProperties, on reading from eventhub.

• Expected behavior: message_id is expected to populate in systemProperties, on reading from eventhub.

• Spark version Spark 3.0.1

• spark-eventhubs artifactId and version com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.18 (Maven central)

I can see the message_id in function app along with body. But, when reading thru spark with the eventhub library its populated as blank {}.

`eventhub_conf = { 'eventhubs.connectionString': spark._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(dbutils.secrets.get(key_vault_name, parameters.get('EVENTHUB_SECRET_NAME'))), 'eventhubs.consumerGroup': 'csl_consumer', "eventhubs.startingPosition": json.dumps(start_position), }

   start_position = {
        "offset": offset,
        "seqNo": sequence_no,
        "enqueuedTime": start_time,
        "isInclusive": True,
    }

    eventhub_conf = {
        'eventhubs.connectionString': spark._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(dbutils.secrets.get(key_vault_name, parameters.get('EVENTHUB_SECRET_NAME'))),
        'eventhubs.consumerGroup': 'yyyy',
        "eventhubs.startingPosition": json.dumps(start_position),
    }

        end_position = {
            "offset": None,  
            "seqNo": -1,  
            "enqueuedTime": end_time,
            "isInclusive": True
        }
        eventhub_conf["eventhubs.endingPosition"] = json.dumps(end_position)

    payload = spark \
        .read \
        .format("eventhubs") \
        .options(**eventhub_conf) \
        .load()

`

[nyaghma]

@jimtodd92 I tried to repro the issue on my side, but my app received the message_id in the systemProperties without any issue. I slightly modified this sample to add message_id to telemetry messages and wrote a simple PySpark app to read messages from the built-in eventhub, using connector version 2.3.18 and spark 3.1.