log4j configurations not added during deployment

[alessandrolulli]

Dear,

I exploited the script to deploy a cluster with a blueprint.

In the provided blueprint i have personalized the content as following:

{
      "yarn-log4j" : {
        "properties_attributes" : { },
        "properties" : {
          "yarn_rm_summary_log_max_backup_size" : "128",
          "yarn_rm_summary_log_number_of_backup_files" : "20",
          "content" : "........... log4j.appender.JSA=org.apache.log4j.**RollingFileAppender** ..........
        }
      }
    }

After the deployment i exported the blueprint of the newly installed cluster. But i obtained the following, where the content is missing:

{
      "yarn-log4j" : {
        "properties_attributes" : { },
        "properties" : {
          "yarn_rm_summary_log_max_backup_size" : "128",
          "yarn_rm_summary_log_number_of_backup_files" : "20"}
      }
    }

Later when exporting the blueprint i obtained the following:

{
      "yarn-log4j" : {
        "properties_attributes" : { },
        "properties" : {
          "yarn_rm_summary_log_max_backup_size" : "128",
          "yarn_rm_summary_log_number_of_backup_files" : "20",
          "content" : ".... log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender ...
        }
      }
    }

I obtained similar behavior with the *-log4j configurations of other services. Is this expected? Do these properties must be manually changed from Ambari after installation?

Let me know if you need additional information.

[alexandruanghel]

Hi @alessandrolulli , not sure if I understand the problem here... So what happened between

After the deployment i exported the blueprint of the newly installed cluster. But i obtained the following, where the content is missing:

and

Later when exporting the blueprint i obtained the following:

Did you do any changes in the configs or it was a simply "5/10 minutes later"?

Normally, when you export a blueprint of a running cluster with ?format=blueprint those "content" variables would appear. And you should be able to use and modify those variables, as long as it doesn't interfere with the NiFi templating as we saw in #66 (inside the "content" you should have stuff with {{ unless carefully escaped).

I have never seen the "content" variables absent from a ?format=blueprint export.

And no, you don't need to set those via Ambari. As long as the "content" variable does not interfere with the Jinja2 templating, you can set whatever you need in there...

So I don't know what additional information I might need as I have a hard time seeing the issue here. Maybe you can give me the "content" variable from your Jinja2 blueprint template?

[alessandrolulli]

Dear @alexandruanghel,

thanks for helping me on this. I discovered the source of this issue. Executing the two sed commands suggested in #66 it is completely removing the "content" line in the blueprint, for instance from the hdfs-log4j configuration.

The two commands are: sed -i '/{{\|{%/d' blueprintest.json sed -i ':a;N;$!ba;s/,\n[[:space:]]*}/}/g' blueprintest.json

For this reason, i deployed a blueprint without the configuration for the log4j variables. This is why part of my configuration for log4j disappered.

I misunderstand such behavior. I think the best solution to solve both this issue and #66 is to not execute the two sed commands and to manually substitute all the variables defined with the {{ syntax.

Thanks

[alessandrolulli]

Do you mind to a better solution with respect to manually substitute the data? Variables are very useful, for instance, in several places in the blueprint it is specified the following: export JAVA_HOME={{java_home}}

Which is the best way to keep {{ variables and avoid #66? Is there a different syntax to define the very same variables?

Thanks

[alexandruanghel]

You should never change the Ambari variables that use {{ because that will be managed by Ambari and it wouldn't be great if we change behaviours expected my Ambari.

For example, your {{java_home}} would work fine initially, but if you will ever change the java location in Ambari using ambari-server setup -j those will fail to be updated by Ambari.

I have possibly found another solution. I haven't tested yet, but in Jinja2, you can change the variable delimited from the default {{ to something else: http://jinja.pocoo.org/docs/2.10/api/#high-level-api

So changing variable_start_string and variable_end_string in your template and the using that new string to define your Ansible/Jinja2 variables, it should work...

[alexandruanghel]

Hmm, this has also been added in Ansible docs in version 2.4 (probably I should check docs more regularly :) ) https://docs.ansible.com/ansible/latest/modules/template_module.html

Also, you can override jinja2 settings by adding a special header to template file. i.e. #jinja2:variable_start_string:'[%', variable_end_string:'%]', trim_blocks: False which changes the variable interpolation markers to [% var %] instead of {{ var }}. This is the best way to prevent evaluation of things that look like, but should not be Jinja2. raw/endraw in Jinja2 will not work as you expect because templates in Ansible are recursively evaluated.

[alessandrolulli]

Wonderful,

so the solution of this and part of #66 is to add something similar to the following: #jinja2:variable_start_string:'[%', variable_end_string:'%]', trim_blocks: False

and to not execute the two sed commands.

I will try in the next days and let you know if this is solving the issue. Thanks

[alexandruanghel]

Did it work? :)

[alessandrolulli]

Unfortunately i have been not able to test this yet. I will update the issue as soon as i complet the test. Thanks