search

xemuliam / docker-nifi

Docker image for Apache NiFi Created from NiFi base image to minimize traffic and deployment time in case of changes should be applied on top of NiFi
23 stars 17 forks source link

Volumes not save flows #10

Open Wilton opened 5 years ago

Wilton commented 5 years ago

Processors are not saved after to and restart the containers, whenever I see the volume database_repository it is modified. I need to create the flows and they stay on the volume. And I have a question regarding clusters volumes, how should I implement them? Should I make a volume for each instance or can I just create a shared volume for all instances?

xemuliam commented 5 years ago

Docker containers should be stateless. Thus all container-related parameters (configuration, processors etc.) should be managed on docker image build stage. But, certainly, you can mount your custom processor as volume

Flow management is a pain-point for NiFi. Here you have 2 options:

Easiest way to work with flow which is stored outside of container:

  1. Create folder with NiFi configuration (use your own local path) 
    docker run --name nifi -d xemuliam/nifi && \
docker cp nifi:/opt/nifi/conf /Users/xemuliam/Projects/Docker/nifi_test && \
docker rm -f nifi && \
ls -la /Users/viacheslav.kalashnikov/Projects/Docker/nifi_test/conf
  2. Run NiFi with folder from previous step mounted as volume docker run --name nifi -d -p 8080 -v /Users/xemuliam/Projects/Docker/nifi_test/conf:/opt/nifi/conf xemuliam/nifi
  3. Make some changes in flow
  4. Drop container docker rm -f nifi
  5. Run fresh NiFi container with mounted configuration folder docker run --name nifi -d -p 8080 -v /Users/xemuliam/Projects/Docker/nifi_test/conf:/opt/nifi/conf xemuliam/nifi
  6. Voila! Your changes are there!
xemuliam commented 5 years ago

Regarding cluster

the best way is to store all node-related information (flow, settings etc.) on separate places. this approach will let you avoid any issues and contention for file-based resources (e.g. flow)

sylnsr commented 4 years ago

@xemuliam I am following your approach. I started with copying only flow.xml.gz and nifi.properties, then starting up a new container mapping just these 2 files. The container will not start however. It has these messages in the log:

sed: cannot rename /opt/nifi/nifi-current/conf/sedTNxUtT: Device or resource busy
replacing target file  /opt/nifi/nifi-current/conf/nifi.properties

and 

Caused by: java.nio.file.FileSystemException: ./conf/flow.xml.gz: Device or resource busy
sylnsr commented 4 years ago

Also .. if the flow data is in flow.xml.gz then whats the point of the flowfile_repository folder?