search

cryptotechguru / EDENomicon

https://nomicon.edenprotocol.io/
8 stars 5 forks source link

Prepare and Launch Development Environments #42

Open Flaxscrip opened 2 years ago

Flaxscrip commented 2 years ago

We need to prepare a full-stack server environment for development.

Flaxscrip commented 2 years ago

Not just dev, but need a staging env too for demo/test/etc.

aiggdev commented 2 years ago

As alternatives to Amazon, GCP, etc. I have used Vultr and Linode in addition to Digital Ocean and would recommend all three.

macterra commented 2 years ago

I'm using Digital Ocean for http://btc.metagamer.org:5000/. Just need a docker node.

Flaxscrip commented 2 years ago

@aiggdev - Would you be our Gatekeeper on this requirement? I'll be the Keymaster setting up the environment as per your recommendations and based on your recent experience setting up a dev env.

Flaxscrip commented 2 years ago

What do we want to call our Test/Demo/UAT environment?

This will likely become a Metatron Testnet. We need a cool name :)

macterra commented 2 years ago

Are you talking about naming the server or assigning a subdomain or something else? There are some good names on this list in the same theme as Metatron. How about Wormwood? :)

Flaxscrip commented 2 years ago

@aiggdev @macterra - This is our next Bounty. Let's prepare our Dev Test environment.

New roles for the Bounty: @flaxscrip - Gatekeeper: will write up server node requirements and needed systems. @aiggdev - Keymaster: will setup the various subsystems and help document the process. @macterra - Co-Keymaster: will setup other subsystems (Tesseract?).

macterra commented 2 years ago

Some initial thoughts on requirements... It would be nice if in order to set up a server node you just had to git clone the metatron-node repo and run docker-compose up -d. Sadly this isn't acceptable because the blockchain config files require credentials and it would be unwise to commit those to github.

The next best thing is to clone the repo, then follow instructions in the README to configure the blockchains. We still have to decide which blockchains we'll want to support initially for a server node. I would recommend both Tesseract and BTC, but BTC requires quite a bit of disk space to sync the blockchain (more than is included in a typical Digital Ocean droplet) so we could start with just supporting Tesseract at first.

The next best thing would be to clone the repo, then run a setup script that automatically generates the blockchain configuration files with new credentials. Even here we have varying levels of quality/effort. We could get a new user and password from the admin, but it would be better to use the newer encrypted RPC credentials feature.

The next best thing would be to use a web UI for setup rather than running a script. I'm curious how umbrel does it. The main point here is that the deliverable for this milestone isn't just a running metatron node, but the repo and documentation to set up a new node repeatedly and securely.

Flaxscrip commented 2 years ago

@macterra - I'm reviewing your requirements here. I'm going to attempt the clone + point the blockchain config to your existing TESS node but with new keys on the server.

Flaxscrip commented 2 years ago

I had some issues exposing external access to the IPFS server packaged within the Metatron release.

I decided to use the "best practices" I could find in launching a publically-accessible IPFS server. I used the IPFS Cluster distribution (https://ipfscluster.io/) to achieve stability and performance.

Note that we DID NOT resolve the keys management issues raised by @macterra above (how does umbrel do it?). We will address this more difficult problem in the Alpha-I release of the "EDEN Proposal" being prepared for the Filecoin Foundation.

Flaxscrip commented 2 years ago

Metatron URL: https://metatron.edenprotocol.io/

IPFS Cluster Gateway URL: https://library.edenprotocol.io/ipfs/ {cid}

Flaxscrip commented 2 years ago

Both servers are now online (Metatron & IPFS Cluster).

For this deliverable, we will be using the IPFS Cluster to host demo data from multiple real-world data producers. The diagram below provides a visual representation of the supply chain; each document will be hosted on IPFS servers.

In a subsequent deliverable, we will be using Metatron to document the "links" between documents and string together a full story.

https://library.edenprotocol.io/ipfs/QmNP1E5jFASqa83rFUS4earJH4fKCy76KMG8tavFHKZYgh?filename=EDEN%20Data.png

Flaxscrip commented 2 years ago

@macterra - I've been using the new IPFS cluster and it's been reliable throughout the week.

I discussed a scope change from "Development" to "Demonstration" server with @realchainlife (Sonia @ Filecoin Foundation) in order to have a stable production quality IPFS server to facilitate the EDEN proposal.

I updated the bounty description to reflect the new goal. Please review and advise on agreement to move forward. This IPFS Cluster will be used to host data from various sources (farm, lab, quality assessments) in support of the story illustrated in the Story Diagram.

Public access gateway: https://library.edenprotocol.io/

This should show the "public" directory on the new IPFS cluster.

The IPFS Cluster runs alongside the Metatron IPFS server; Metatron remains on port 5001 while the IPFS Cluster API is on port 5002 and Gateway 8082.

Admin functionality is available by ssh (either tunnel or shell to server). I created 2 aliases to facilitate sending commands to the 2 IPFS containers.

alias metatron = 'ipfs --api /ip4/127.0.0.1/tcp/5001
alias eden = 'ipfs --api /ip4/127.0.0.1/tcp/5002'

I use an ssh tunnel from my local development laptop and connect a local ipfs-desktop to the tunneled server. ssh -N -L 5002:localhost:5002 {ssh credentials here}

The cluster gateway is exposed using nginx + certbot.

Sarah-lee123 commented 2 years ago

Both servers are now online (Metatron & IPFS Cluster).

For this deliverable, we will be using the IPFS Cluster to host demo data from multiple real-world data producers. The diagram below provides a visual representation of the supply chain; each document will be hosted on IPFS servers.

In a subsequent deliverable, we will be using Metatron to document the "links" between documents and string together a full story.

https://library.edenprotocol.io/ipfs/QmNP1E5jFASqa83rFUS4earJH4fKCy76KMG8tavFHKZYgh?filename=EDEN%20Data.png

looks good. It opened right up for me.

macterra commented 2 years ago

Looks good!

Flaxscrip commented 2 years ago

@macterra - Thank you for the review and your help connecting the servers. It looks really nice now. I'll restart my Tesseract miner, create a new wallet, and send some funds to the EDEN node. Good times ahead!

I will connect with our Treasurer for invoicing. While it's been mainly you and I involved, @aiggdev has been covering quite a bit for me at work, so I'd like to keep the same payment rules, 1/3 to each of our FIL addresses, as with our prior bounty.

Flaxscrip commented 2 years ago

I heard back from Filecoin Foundation. There is a new process whereby I scheduled a deliverable review with our Treasurer and a new FIL resource called Erin. The review is scheduled for Sept 1st.

I will introduce Erin to the Nomicon process at a high level, honing in to Bounty #42. We will review highlights and bounty changes over the development period. We will review the 2 deliverable links: IPFS Cluster and Metatron.

Ahead of the meeting, I propose to add a section to the Bounty page for links to deliverables. This will make it easier to use the Bounty page as an "agenda" for the review session.