Too much logging from the app

[raviteja2103]

Hello Abris team, We're using your API in our streaming app. It's really helpful, thanks for it. We're using "za.co.absa:abris_2.11:3.0.3".

However, we're seeing huge logs being stored in temp/ location. We dont want to change at the Spark basic config level since its the only application which is causing this issue and others are fine. Can you help us with how to suppress these logs at the application level?

99% of the data is this:

20/02/12 18:53:52 INFO serializers.KafkaAvroDeserializerConfig: KafkaAvroDeserializerConfig values: schema.registry.url = [http://caution-pc-sr.sitescout.com:8091] basic.auth.user.info = [hidden] auto.register.schemas = true max.schemas.per.subject = 1000 basic.auth.credentials.source = URL schema.registry.basic.auth.user.info = [hidden] specific.avro.reader = false value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy

482 [GB] /tmp/logs/x/logs/application_1580249944747_7589 483 [GB] /tmp/logs/x/logs/application_1580249944747_7778 486 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4377 488 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7204 559 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4172 563 [GB] /tmp/logs/x/logs/application_1580249944747_6584 568 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_6157 573 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7767 577 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4299 577 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4520 577 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7960 579 [GB] /tmp/logs/x/logs/application_1580249944747_6489 579 [GB] /tmp/logs/x/logs/application_1580249944747_7133 582 [GB] /tmp/logs/x/logs/application_1580249944747_7567 595 [GB] /tmp/logs/x/logs/application_1580249944747_4432 597 [GB] /tmp/logs/x/logs/application_1580249944747_5642 597 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_6914 598 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_6856 599 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4348 600 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4919 600 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_6500 601 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_6796 602 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7229 602 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7297 602 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7442 603 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_3780 603 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_3810 603 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7509 604 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7366 607 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4642 607 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5042 607 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7631 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4648 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4657 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4698 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4711 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4719 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4763 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5684 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5703 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5708 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5749 608 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_6691 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4224 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5767 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5776 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5817 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5828 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5842 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5889 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5896 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5916 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_5955 609 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_6555 610 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4233 610 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4268 610 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4289 610 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4357 611 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4106 611 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4441 611 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4827 611 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_4881 611 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7833 611 [GB] /tmp/logs/bdbi_preprod_dev/logs/application_1580249944747_7871 612 [GB] /tmp/logs/x/logs/application_1580249944747_4818 612 [GB] /tmp/logs/x/logs/application_1580249944747_4846 612 [GB] /tmp/logs/x/logs/application_1580249944747_6173 612 [GB] /tmp/logs/x/logs/application_1580249944747_6509

[raviteja2103]

Looks like the issue is happening because of this debug statements here: https://github.com/AbsaOSS/ABRiS/blob/master/src/main/scala/za/co/absa/abris/avro/read/confluent/SchemaManager.scala

[raviteja2103]

The issue got resolved when I used the latest version 3.1.1