Simplified setup for self-hosting?

[poVoq]

I would like to try setting up a small version of this for private use on my own server, but even the docker version seems excessively complex with three redundant elastic search instances and such.

Is there a simpler way (less performant of course) to just run it as a Python app with MariaDB only? I don't mind using Docker in general, but something simple with standard dependencies from the Debian Bullseye repositories would be best.

Thanks!

[qlands]

The docker-compose file at https://raw.githubusercontent.com/qlands/FormShare/master-2.0/docker_compose/docker-compose.yml has one MySQL server and two nodes of Elasticsearch (ES). However, you can have just one node of ES if this is for testing purposes.

FormShare uses MySQL. Maria DB even though is almost a drop-down replacement of MySQL is not supported. You also don't need to use MySQL in the docker file. You can use the MySQL of your own Debian Bullseye distro and change the variables: MYSQL_HOST_NAME, MYSQL_USER_NAME, and MYSQL_USER_PASSWORD on the docker file.

The following compose uses just one ES node:

version: '3'
services:
  fsmysql_20220110:
    image: mysql:8.0.23
    command: --default-authentication-plugin=mysql_native_password
    container_name: fs_mysql_20220110
    cap_add:
      - SYS_NICE
    environment:
      MYSQL_ROOT_PASSWORD: my_secure_password
    volumes:
      - /opt/formshare/mysql:/var/lib/mysql
    networks:
      fsnet:
        ipv4_address: 172.28.1.5

  fses20220110n01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.2
    container_name: fses20220110n01
    environment:
      - network.host=172.28.1.1
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - transport.host=127.0.0.1
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /opt/formshare/elasticsearch/esdata:/usr/share/elasticsearch/data
    networks:
      fsnet:
        ipv4_address: 172.28.1.1

  formshare_20220110:
    image: qlands/formshare2:20220110
    container_name: formshare_20220110
    environment:
      MYSQL_HOST_NAME: 172.28.1.5
      MYSQL_USER_NAME: root
      MYSQL_USER_PASSWORD: my_secure_password
      FORMSHARE_ADMIN_USER: admin
      FORMSHARE_ADMIN_EMAIL: admin@myserver.com
      FORMSHARE_ADMIN_PASSWORD: my_secure_password
      ELASTIC_SEARCH_HOST: 172.28.1.1
      ELASTIC_SEARCH_PORT: 9200
      FORMSHARE_HOST: 172.28.1.4
      FORMSHARE_PORT: 5900
      FORWARDED_ALLOW_IP: 172.28.1.4
      WAIT_HOSTS_TIMEOUT: 360
      WAIT_HOSTS: 172.28.1.5:3306, 172.28.1.1:9200
    volumes:
      - /opt/formshare/repository:/opt/formshare_repository
      - /opt/formshare/log:/opt/formshare_log
      - /opt/formshare/celery:/opt/formshare_celery
      - /opt/formshare/config:/opt/formshare_config
      - /opt/formshare/plugins:/opt/formshare_plugins
      - /opt/formshare/formshare_odata_webapps:/opt/formshare_odata_webapps
      - /opt/formshare/mosquitto:/etc/mosquitto/conf.d/
    ports:
      - 5900:5900
      - 9001:9001
    networks:
      fsnet: 
        ipv4_address: 172.28.1.4
networks:
  fsnet:
     ipam:
        driver: default
        config:
          - subnet: 172.28.0.0/16

You could also use ElasticSearch from the Debian Bullseye repository and just have the section for FormShare in the docker file. But if you use ES from Debian Bullseye repository then you need to be sure that it is configured correctly and that it runs properly (see section "Set enough memory for Elasticsearch" in the readme).

But at the end, FormShare needs both: ES and MySQL

[qlands]

How did it go? Can I close the issue?

[poVoq]

Still working on it. But I'll close it. Thanks for the help.

I am trying it with a local mariadb and elastic search instance. My problem right now is that elastic search uses too much memory, not too little ;)

[qlands]

Yes. We understand. ES is heavy but it powers a lot of complex operations. For example, Geo-clustering when you have millions of GPS points. That's why we have the https://formshare.org/ . Because many small organizations cannot have the resources or technical staff to have their own installation. formshare.org is complete but generic. You cannot have custom plugins loaded there. However, formshare.org will have soon OAuth2 Server and Client capabilities.

[poVoq]

Depending on the complexity of your elastic search usage, this might be a drop-in replacements that needs much less resources: https://github.com/prabhatsharma/zinc

[qlands]

Thanks for the link. We will definitely will look at zinc and its capabilities.

[poVoq]

I am trying to run it with a local MariaDB instance, but I get the following error message from the Formshare docker container:

File "/opt/formshare_env/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 513, in cmd_query self._cmysql.query(query, _mysql_connector.MySQLInterfaceError: Unknown collation: 'utf8mb4_0900_ai_ci'

and a bit later again:

sqlalchemy.exc.DatabaseError: (mysql.connector.errors.DatabaseError) 1273 (HY000): Unknown collation: 'utf8mb4_0900_ai_ci'

I wasn't sure how to create the database, so I made one like this CREATE DATABASE IF NOT EXISTS formshare CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; and granted all privileges to the configured user.

Any idea what I should do better?

The connection to the local MariaDB an the local Elastic search seems to work when running the docker container in the host network.

[qlands]

FormShare does not support MariaDB. We understand that MariaDB should be a drop-down replacement for MySQL but we don't test FormShare on MariaDB and we cannot look into compatibility problems between MySQL an Maria DB.

For this particular error, we don't use such collection (utf8mb4_0900_ai_ci) in the code. FormShare database and the databases that FormShare creates are in "utf8mb4_unicode_ci". If you getting such error is that maybe SQLAlchemy is using a wrong collection with your Maria DB.

Please run FormShare using MySQL 8.X. We had MariaDB in the installation steps. We removed it now. Sorry for the confusion.

[poVoq]

Any way to make this work without hard-coding IP addresses in the docker network? I suspect this is the reason why I can't get this to work with Podman (no error in the logs and I always get a 502 Bad Gateway error).

For some reason the logs also claim that it listens on 127.0.0.1:5900, which will not work in a container environment where it should listen on 0.0.0.0:5900 instead.

[qlands]

You can change the docker-compose file (docker_compose/docker-compose.yml) so that the IP address is not hardcoded. Then modify the file docker_files/formshare/docker_files/run_server.sh s to specify the parameter --formshare_host to not come from environmental variables but from the current IP of the server running such script.