search

microcks / microcks

The open source, cloud native tool for API Mocking and Testing. Microcks is a Cloud Native Computing Foundation sandbox project 🚀
https://microcks.io
Apache License 2.0
1.41k stars 222 forks source link

Not able to mock async api with Kafka + Schema Registry #1411

Open brunodomenici opened 1 day ago

brunodomenici commented 1 day ago

Describe the bug

I have a log message Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done:

2024-11-14 16:09:02,863 INFO  [io.git.mic.min.asy.AsyncMinionApp] (Quarkus Main Thread) Microcks Keycloak server url {http://localhost:18080} and realm {microcks}
2024-11-14 16:09:02,863 INFO  [io.git.mic.min.asy.AsyncMinionApp] (Quarkus Main Thread) Use locally defined Keycloak Auth URL: Optional[http://localhost:18080]
2024-11-14 16:09:02,930 INFO  [io.git.mic.min.asy.AsyncMinionApp] (Quarkus Main Thread) Authentication to Keycloak server succeed!
2024-11-14 16:09:02,997 INFO  [io.git.mic.min.asy.AsyncMinionApp] (Quarkus Main Thread) Found 2 candidate operations in NAMESPACES - v1
2024-11-14 16:09:03,042 INFO  [io.git.mic.min.asy.SchemaRegistry] (Quarkus Main Thread) Updating schema registry for 'NAMESPACES - v1' with 1 entries
2024-11-14 16:09:03,050 INFO  [io.git.mic.min.asy.SchemaRegistry] (Quarkus Main Thread) Updating schema registry for 'NAMESPACES - v1' with 1 entries
2024-11-14 16:09:03,051 INFO  [io.git.mic.min.asy.AsyncMinionApp] (Quarkus Main Thread) Starting scheduling of all producer jobs...
2024-11-14 16:09:03,051 INFO  [io.git.mic.min.asy.ProducerScheduler] (Quarkus Main Thread) Scheduling a new Producer Job at frequency 3
2024-11-14 16:09:03,053 INFO  [io.git.mic.min.asy.ProducerScheduler] (Quarkus Main Thread) Scheduling a new Producer Job at frequency 10
2024-11-14 16:09:03,053 INFO  [io.git.mic.min.asy.ProducerScheduler] (Quarkus Main Thread) Scheduling a new Producer Job at frequency 30
2024-11-14 16:09:03,059 INFO  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-2) Producing async mock messages for frequency: 3
2024-11-14 16:09:03,059 INFO  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-3) Producing async mock messages for frequency: 10
2024-11-14 16:09:03,059 INFO  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-4) Producing async mock messages for frequency: 30
2024-11-14 16:09:03,104 INFO  [io.con.kaf.ser.KafkaAvroSerializerConfig] (QuarkusQuartzScheduler_Worker-2) KafkaAvroSerializerConfig values: 
        auto.register.schemas = true
        avro.reflection.allow.null = false
        avro.remove.java.properties = false
        avro.use.logical.type.converters = false
        basic.auth.credentials.source = USER_INFO
        basic.auth.user.info = [hidden]
        bearer.auth.credentials.source = STATIC_TOKEN
        bearer.auth.token = [hidden]
        key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy
        latest.compatibility.strict = true
        max.schemas.per.subject = 1000
        proxy.host = 
        proxy.port = -1
        schema.reflection = false
        schema.registry.basic.auth.user.info = [hidden]
        schema.registry.ssl.cipher.suites = null
        schema.registry.ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        schema.registry.ssl.endpoint.identification.algorithm = https
        schema.registry.ssl.engine.factory.class = null
        schema.registry.ssl.key.password = null
        schema.registry.ssl.keymanager.algorithm = SunX509
        schema.registry.ssl.keystore.certificate.chain = null
        schema.registry.ssl.keystore.key = null
        schema.registry.ssl.keystore.location = null
        schema.registry.ssl.keystore.password = null
        schema.registry.ssl.keystore.type = JKS
        schema.registry.ssl.protocol = TLSv1.3
        schema.registry.ssl.provider = null
        schema.registry.ssl.secure.random.implementation = null
        schema.registry.ssl.trustmanager.algorithm = PKIX
        schema.registry.ssl.truststore.certificates = null
        schema.registry.ssl.truststore.location = null
        schema.registry.ssl.truststore.password = null
        schema.registry.ssl.truststore.type = JKS
        schema.registry.url = [https://xxx]
        use.latest.version = false
        value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

2024-11-14 16:09:03,115 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-2) Failed finding a suitable Avro schema for the 'SUBSCRIBE NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:03,116 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-2) Failed finding a suitable Avro schema for the 'SUBSCRIBE NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:03,116 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-2) Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:03,117 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-2) Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:03,117 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-2) Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:03,126 INFO  [io.quarkus] (Quarkus Main Thread) microcks-async-minion 1.10.2-SNAPSHOT on JVM (powered by Quarkus 3.8.4) started in 4.169s. Listening on: http://localhost:8081
2024-11-14 16:09:03,127 INFO  [io.quarkus] (Quarkus Main Thread) Profile  activated. Live Coding activated.
2024-11-14 16:09:03,128 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [amazon-sns, amazon-sqs, apicurio-registry-avro, cdi, confluent-registry-avro, google-cloud-pubsub, kafka-client, micrometer, quartz, resteasy, resteasy-client, resteasy-jackson, scheduler, smallrye-context-propagation, smallrye-health, smallrye-reactive-messaging, smallrye-reactive-messaging-kafka, vertx, websockets, websockets-client]
2024-11-14 16:09:06,054 INFO  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-5) Producing async mock messages for frequency: 3
2024-11-14 16:09:06,055 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-5) Failed finding a suitable Avro schema for the 'SUBSCRIBE NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:06,056 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-5) Failed finding a suitable Avro schema for the 'SUBSCRIBE NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:06,057 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-5) Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:06,060 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-5) Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:06,061 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-5) Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:09,057 INFO  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-6) Producing async mock messages for frequency: 3
2024-11-14 16:09:09,058 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-6) Failed finding a suitable Avro schema for the 'SUBSCRIBE NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:09,058 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-6) Failed finding a suitable Avro schema for the 'SUBSCRIBE NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:09,059 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-6) Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:09,059 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-6) Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done.
2024-11-14 16:09:09,060 WARN  [io.git.mic.min.asy.pro.ProducerManager] (QuarkusQuartzScheduler_Worker-6) Failed finding a suitable Avro schema for the 'PUBLISH NAMESPACES-V1' operation. No publication done.

Expected behavior

Mock messages published in the Kafka topic

Actual behavior

Warning messages saying there is no schema.

How to Reproduce?

No response

Microcks version or git rev

microcks:1.10.1 , async-minion:master

Install method (docker-compose, helm chart, operator, docker-desktop extension,...)

docker

Additional information

I did some debugging and there no operations returning from Microcks APIs:

https://github.com/microcks/microcks/blob/59e0f49f00ffbf2dec7cbb7004416cc1f4552b41/minions/async/src/main/java/io/github/microcks/minion/async/producer/ProducerManager.java#L155-L158

Resources API:

GET http://localhost:8080/api/resources/service/67360c3abd51570da5e1cdb4

HTTP/1.1 200 
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: *
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
Content-Type: application/json
Transfer-Encoding: chunked
Date: Thu, 14 Nov 2024 15:18:43 GMT
Keep-Alive: timeout=60
Connection: keep-alive

[
  {
    "id": "67360c3abd51570da5e1cdb5",
    "name": "NAMESPACES-v1.yaml",
    "content": "xxx",
    "type": "ASYNC_API_SPEC",
    "serviceId": "67360c3abd51570da5e1cdb4",
    "sourceArtifact": "namespace.yaml",
    "mainArtifact": true
  }
]

I'm not sure why.

However, I see operations in Services API:

GET http://localhost:8080/api/services/67360c3abd51570da5e1cdb4

HTTP/1.1 200 
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: *
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
Content-Type: application/json
Transfer-Encoding: chunked
Date: Thu, 14 Nov 2024 15:20:51 GMT
Keep-Alive: timeout=60
Connection: keep-alive

{
  "service": {
    "id": "67360c3abd51570da5e1cdb4",
    "name": "NAMESPACES",
    "version": "v1",
    "type": "EVENT",
    "metadata": {
      "createdOn": 1731595322236,
      "lastUpdate": 1731595322237
    },
    "sourceArtifact": "namespace.yaml",
    "operations": [
      {
        "name": "PUBLISH NAMESPACES-V1",
        "method": "PUBLISH",
        "bindings": {
          "KAFKA": {
            "type": "KAFKA",
            "persistent": false
          }
        },
        "defaultDelay": 3,
        "resourcePaths": [
          "xxx"
        ]
      },
      {
        "name": "SUBSCRIBE NAMESPACES-V1",
        "method": "SUBSCRIBE",
        "bindings": {
          "KAFKA": {
            "type": "KAFKA",
            "persistent": false
          }
        },
        "defaultDelay": 3,
        "resourcePaths": [
          "xxx"
        ]
      }
    ]
  },
  "messagesMap": xx
    ]
  }
}
github-actions[bot] commented 1 day ago

👋 @brunodomenici

Welcome to the Microcks community! 💖

Thanks and congrats 🎉 for opening your first issue here! Be sure to follow the issue template or please update it accordingly.

📢 If you're using Microcks in your organization, please add your company name to this list. 🙏 It really helps the project to gain momentum and credibility. It's a small contribution back to the project with a big impact.

If you need to know why and how to add yourself to the list, please read the blog post "Join the Microcks Adopters list and Empower the vibrant open source Community 🙌"

Hope you have a great time there!

🌟 ~~~~~ 🌟

📢 If you like Microcks, please ⭐ star ⭐ our repo to support it!

🙏 It really helps the project to gain momentum and credibility. It's a small contribution back to the project with a big impact.