gitflowpersistanceprovider doesnot show the gitflow in gitlab

Sanjanare commented 5 years ago

@michalklempa Hi , i have tried the dockerfile for gitlab persistance provider , i am running it on kubernetes

Here is the docker image i am using

FROM michalklempa/nifi-registry

ENV FLOW_PROVIDER=git \
    GIT_REMOTE_URL=https://*************/nifi-registry-git.git \
    GIT_CHECKOUT_BRANCH=  \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER=amx-bot \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD=******** \
    GIT_CONFIG_USER_NAME=amx-bot \
    GIT_CONFIG_USER_EMAIL=amx-bot@*******

i get below logs after running the dockerfile

Loading...FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD is set, trying to set git credential helper for HTTPS password
git config --global credential.${GIT_REMOTE_URL}.helper '!f() { sleep 1; echo -e "username=${FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER}\npassword=*****"; }; f'
Found git remote: https:/********/nifi-registry-git.git, cloning into: /opt/nifi-registry/flow-storage, with remote:  and branch: 
git clone -o $FLOW_PROVIDER_GIT_REMOTE_TO_PUSH -b $GIT_CHECKOUT_BRANCH $GIT_REMOTE_URL $FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY
Cloning into '/opt/nifi-registry/flow-storage'...
Found key: GIT_CONFIG_USER_EMAIL for git config
Setting git config: user.email=**********
git config -f ${FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY}/.git/config ${KEY} '${VALUE}'
Found key: GIT_CONFIG_USER_NAME for git config
Setting git config: user.name=amx-bot
git config -f ${FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY}/.git/config ${KEY} '${VALUE}'
NiFi-Registry running with PID 32.
tail: can't open '/opt/nifi-registry/nifi-registry-0.4.0/logs/nifi-registry-app.log': No such file or directory

Java home: /usr/lib/jvm/java-1.8-openjdk
NiFi Registry home: /opt/nifi-registry/nifi-registry-0.4.0

Bootstrap Config File: /opt/nifi-registry/nifi-registry-0.4.0/conf/bootstrap.conf

And the nifi-registry starts initializing.

When i give the http://localhost:18080 in nifi to connect to registry, it is able to see the buckets in nifi-registry , after adding the version control to the nifi i can see the commit changes in the nifi-registry but i am unable to see this changes in the gitlab repo.

it is am empty repository and even after commiting changes to nifi-regsitry it is still empty as i understand i should be able to see the commits in gitlab repo ? please let me know if i am missing anything . Thanks

michalklempa commented 5 years ago

Hi @Sanjanare, I was trying this against my own private gitlab instance and I was able to commit & push changes.

I guess, maybe you left out the branch of remote git repository?

ENV FLOW_PROVIDER=git \
    GIT_REMOTE_URL=https://*************/nifi-registry-git.git \
    `GIT_CHECKOUT_BRANCH= \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER=amx-bot \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD=******** \
    GIT_CONFIG_USER_NAME=amx-bot \
    GIT_CONFIG_USER_EMAIL=amx-bot@*******

This value:

    `GIT_CHECKOUT_BRANCH= \

Should not be empty. Try master branch.

Full example:

FROM michalklempa/nifi-registry

ENV FLOW_PROVIDER=git \
    GIT_REMOTE_URL=https://*************/nifi-registry-git.git \
    GIT_CHECKOUT_BRANCH=master  \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER=amx-bot \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD=******** \
    GIT_CONFIG_USER_NAME=amx-bot \
    GIT_CONFIG_USER_EMAIL=amx-bot@*******

And you do not have to create child Dockerfile for this to try. Just use docker run command:

docker run --name nifi-registry \
      -p 18080:18080 \
      -e 'FLOW_PROVIDER=git' \
      -e 'GIT_REMOTE_URL=https://*************/nifi-registry-git.git' \
      -e 'GIT_CHECKOUT_BRANCH=master' \
      -e 'FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY=/opt/nifi-registry/flow-storage-git' \
      -e 'FLOW_PROVIDER_GIT_REMOTE_TO_PUSH=origin' \
      -e 'FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER=amx-bot ' \
      -e 'FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD=********' \
      -e 'GIT_CONFIG_USER_NAME=amx-bot' \
      -e 'GIT_CONFIG_USER_EMAIL=amx-bot@********'' \
      -d \
      michalklempa/nifi-registry:latest

Give it a try and let me know.

Sanjanare commented 5 years ago

@michalklempa i will try and let you know thank you :) . I am creating a child Dockerfile since it is in ci/cd process and we are using helm files to deploy

Sanjanare commented 5 years ago

@michalklempa when i add GIT_CHECKOUT_BRANCH=master it gives out below error

docker run -p 18080:18080 nifi-registry:latest
FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD is set, trying to set git credential helper for HTTPS password
git config --global credential.${GIT_REMOTE_URL}.helper '!f() { sleep 1; echo -e "username=${FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER}\npassword=*****"; }; f'
Found git remote: https://**********/nifi-registry-git.git, cloning into: /opt/nifi-registry/flow-storage, with remote: master and branch:
git clone -o $FLOW_PROVIDER_GIT_REMOTE_TO_PUSH -b $GIT_CHECKOUT_BRANCH $GIT_REMOTE_URL $FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY
fatal: Too many arguments.

usage: git clone [<options>] [--] <repo> [<dir>]

if i remove master and try it works fine and initializes nifi-registry

Sanjanare commented 5 years ago

@michalklempa nvm it worked after adding all the env variables like below:


FROM michalklempa/nifi-registry

ENV FLOW_PROVIDER=git \
    GIT_REMOTE_URL=https://*********/nifi-registry-git.git \
    GIT_CHECKOUT_BRANCH=master \
    FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY=/opt/nifi-registry/flow-storage-git \
    FLOW_PROVIDER_GIT_REMOTE_TO_PUSH=origin \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER=amx-bot \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD=****** \
    GIT_CONFIG_USER_NAME=amx-bot \
    GIT_CONFIG_USER_EMAIL=amx-bot@*******

Sanjanare commented 5 years ago

@michalklempa i am trying to configure ldap properties in nifi registry and i see authorizations.xml is by default configured to use file-user-group-provider but i want to configure LdapUserGroupProvider , i could not find the variables for LdapUserGroupProvider in the documentation https://github.com/michalklempa/docker-nifi-registry/#standalone-instance-ldap can you tell me how to configure those ?

Thanks you so much for you help in before task

michalklempa commented 5 years ago

@Sanjanare LdapUserGroupProvider - you point to changing conf/authorizers.xml?

If so, I am sorry to admit, that these configuration options are not inserted from environmental variables. There are too many options in my opinion, to handle them using env vars.

If you can provide me an example of kind-of working authorizers.xml file, I can maybe think of variable names and incorporate the changes needed. You can reach me out using email michal.klempa@gmail.com. Also, if you are willing to contribute such a change, lets go with pull request.

In the meantime, in k8s, I think there is an option to provide mounted volume and/or ConfigMap (https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) - so you could be able to provide the authorizers.xml as a whole file and omit any changes to this file by Docker image. To do so, mount your authorizers.xml file into k8s pod on location:

/opt/nifi-registry/nifi-registry-0.4.0/conf/authorizers.xml

And avoid setting INITIAL_ADMIN_IDENTITY environment variable (as this decides, whether start.sh tries to render template into the file):

Snippet from start.sh:

if [[ -n "${INITIAL_ADMIN_IDENTITY}" ]]; then
    /usr/local/bin/dockerize -template ${PROJECT_TEMPLATE_DIR}/authorizers.xml.gotemplate:${PROJECT_CONF_DIR}/authorizers.xml
fi

Virtually any configuration file can be provided as ConfigMap and/or mounted file/volume. See https://github.com/michalklempa/docker-nifi-registry/blob/develop/sh/start.sh#L74 There you find consecutive if-statements - each for one configuration file. Just avoid setting the env vars which trigger templating and you can provide own configuration files.

See this example for mounting: https://github.com/michalklempa/docker-nifi-registry#providing-configuration-by-mounting-files Same can be achieved in k8s.

Hope this helps.

Sanjanare commented 5 years ago

@michalklempa we didnot make changes to the authorizers.xml file , we dropped from this plan , but will send u PR if we add anything . ,thank you so much :)

Sanjanare commented 5 years ago

@michalklempa since registry is working now , i will get authorizers.xml set of Users List, i will reach out to you in email after i setup . Thanks for ur help

michalklempa commented 5 years ago

@Sanjanare may we close this issue?

michalklempa / docker-nifi-registry

gitflowpersistanceprovider doesnot show the gitflow in gitlab #14