How to configure replica for the mahuta docker machine

[nlko]

Hello,

I didn't find in the documentation (nor the http-api code) how to integrate the mahuta docker machine with an ipfs-cluster.

Is there an environnent variable I can pass to the mahuta docker machine for that purpose ?

According to #31, there use to be an example but I didn't manage either to make it work.

Have a nice day,

Nicolas

[gjeanmart]

Hi Nicolas, Thanks for taking time to fill this issue. I have to admit I have omitted the configuration part for the HTTP-API. But this should be fixed now on the development branch.

If you want to try out before the release, you can use Docker image gjeanmart/mahuta:0.3.8-SNAPSHOT or gjeanmart/mahuta:snapshot.

I explained the details of the Docker configuration for Mahuta HTTP-API into this doc.

To summarise, you can use either environment variables or an external YAML file passed to the container.

Feel free to contact me if you have any issues with this.

[nlko]

Hello,

I'm not able to test it right now but I'll sure do it next week.

The documentation sounds good to me.

Thanks for the fix anyway,

Nicolas

[nlko]

Hello,

From what I see :

MAHUTA_IPFS_REPLICAIPFS_0_MULTIADDRESS: /dnsaddr/ipfs.infura.io/tcp/5001/https

I understand that it references an ipfs instance and not an ipfs-cluster instance.

Am I right ? I thought mahuta was using ipfs-cluster for replication ?

[gjeanmart]

Hey,

You can use one or more IPFS-Cluster node as a replica or directly another IPFS nodes (like I did with Infura). The pinning (IPFS-Cluster or IPFS) will happen asynchronously.

In theory (I'm having hard time simulating this with docker and IPFS-Cluster), you should be able to use the following env variables:

• MAHUTA_IPFS_REPLICACLUSTERIPFS_0_HOST
• MAHUTA_IPFS_REPLICACLUSTERIPFS_0_PORT

this is converted to the spring-boot yaml properties:

mahuta:
  ipfs:
    host: localhost
    port: 5001
    timeout:
       read: 5000
       write: 2000
    threadPool: 10
    replicaIPFS:
    - host: 
      port: 
    replicaClusterIPFS:
    - host: 
      port: 

Hope this help!

[nlko]

Hi,

Still no luck. I tried

      MAHUTA_IPFS_REPLICACLUSTERIPFS_0_HOST: cluster0
      MAHUTA_IPFS_REPLICACLUSTERIPFS_0_PORT: 9094

But it seems completely ignored by mahuta. That is the file is pinned at an ipfs level but completely ignored by ipfs-cluster. Am I using the correct environment variables?

As you can see in the following I'm able to pin a file using ipfs-cluster directly (outside mahuta) and it replicates on all the clusters. Mahuta still only pin on an ipfs level as I understand from the result and the log.

$ curl -X POST     'http://localhost:8040/mahuta/index'     -H 'content-type: application/json'     -d '{"content":"# Hello world,\n this is my first file stored on **IPFS**","indexName":"articles","indexDocId":"hello_world","contentType":"text/markdown","index_fields":{"title":"Hello world", "author":"Gregoire Jeanmart","votes":10,"date_created":1518700549,"tags":["general"]}}'
 {"indexName":"articles","indexDocId":"hello_world","contentId":"Qmd6VkHiLbLPncVQiewQe3SBP8rrG96HTkYkLbMzMe6tP2","contentType":"text/markdown","pinned":true,"indexFields":null,"status":"SUCCESS"}

# Mahuta logs:
# mahuta_1         | [2019-07-17 15:03:04,718] [DEBUG] [n.c.m.c.s.s.i.IPFSService IPFSService.java:187] Pin CID Qmd6VkHiLbLPncVQiewQe3SBP8rrG96HTkYkLbMzMe6tP2 on IPFS
# mahuta_1         | [2019-07-17 15:03:04,735] [DEBUG] [n.c.m.c.s.s.i.IPFSService IPFSService.java:193] CID Qmd6VkHiLbLPncVQiewQe3SBP8rrG96HTkYkLbMzMe6tP2 pinned on IPFS

$ docker-compose exec ipfs0 ipfs pin ls | grep Qmd6VkHiLbLPncVQiewQe3SBP8rrG96HTkYkLbMzMe6tP2

Qmd6VkHiLbLPncVQiewQe3SBP8rrG96HTkYkLbMzMe6tP2 recursive

$ docker-compose exec cluster2 ipfs-cluster-ctl pin ls 
# returns nothing

$ docker-compose exec cluster0 ipfs-cluster-ctl pin add Qmd6VkHiLbLPncVQiewQe3SBP8rrG96HTkYkLbMzMe6tP2
Qmd6VkHiLbLPncVQiewQe3SBP8rrG96HTkYkLbMzMe6tP2 :
    > 9bfa590c095b    : PINNED | 2019-07-17T14:59:39.76276035Z
    > a4ba15c3cbb3    : PINNED | 2019-07-17T14:59:39.860652466Z
    > 74f59845f8af    : PINNED | 2019-07-17T14:59:39.823451005Z

$ docker-compose exec cluster2 ipfs-cluster-ctl pin ls
Qmd6VkHiLbLPncVQiewQe3SBP8rrG96HTkYkLbMzMe6tP2 |  | PIN | Repl. Factor: -1 | Allocations: [everywhere] | Recursive

I was expecting the ipfs-cluster-ctl pin ls command to return something after the POST to mahuta (meaning that mahuta has dealt with ipfs-cluster).

Thanks for helping.

[gjeanmart]

Alright, I have been able to test everything out with docker-compose (you can find the sample file here).

Basically, two issues:

1. Typo with property keys (between the yml file and in the java code). it is MAHUTA_IPFS_REPLICAIPFSCLUSTER_0_..... and not MAHUTA_IPFS_REPLICACLUSTERIPFS_0_.....

MAHUTA_IPFS_REPLICAIPFSCLUSTER_0_HOST: ipfs-cluster0
MAHUTA_IPFS_REPLICAIPFSCLUSTER_0_PORT: 9094

2. The HTTP query to pin a file to a cluster was prepared but not sent :-|

Please have a look to gjeanmart/mahuta:snapshot and let me know if it works, if so, I will merge to master and release a new version.

Demo: https://imgur.com/3dkuIzn

[nlko]

Yes! it works for me too now!

Thanks!

[gjeanmart]

Awesome ! I'll close the issue when it's release!

[gjeanmart]

I'm closing the issue. Fix in the release 0.3.8

[gjeanmart]

Thanks again for your help with this ;)