search

cardano-foundation / developer-portal

The Cardano Developer Portal
https://developers.cardano.org
MIT License
375 stars 901 forks source link

Revamp Stake Pool Section: new structure proposal #737

Closed lodl closed 1 year ago

lodl commented 2 years ago

Because of a better overview in the Stake Pool Section, I would suggest the following new outline of the menu structure for creating and maintaining a Stake Pool in Testnet.

It is a proposal that everything listed in the current Stake Pool Operator section is replaced by a new and clearer structure.

Overview

This is the first draft of this idea, please let me know what you think.

Thanks, @sanskys for your issue #641 I hope my proposal fits your ideas.

os11k commented 1 year ago

i think it should be a separate guide and all chapters of this guide should be re-written/adapted for docker.

Technically, you don't need to rewrite everything for Docker; all that applies to a normal node will apply to a Docker node too. There is no difference in config files or whatever; only Cardano CLI commands should be prefixed with docker exec -ti ....

ok..dint know that. I had in mind that there are several differences like how Prometheus/Grafana work and also security considerations. Lets wait for what others say.

In my example, Docker uses the host network, so it means that the Docker container will be plugged into the host network, so no NAT and the same iptables FW rules will apply as if you were securing a normal Linux machine. Only Docker will have some additional iptables chains, but those can be ignored. There are some security implications, but you would really need to try to do something wrong to create security holes in your Docker setup.

rdlrt commented 1 year ago

all that applies to a normal node will apply to a Docker node too.

There could certainly be a large gap in usage if SPO isnt a regular docker user, there is additional layer of basic troubleshooting (as often docker users end up with posts like 'does not work' on forum/SE/GH/etc without as much as trying to check container status/errors/verifying network ports/mounts/handling restarts/checking logs/querying tip/interacting with node using socket or port/etc) - so if included, it should certainly not be done without careful admonitions (as an addendum to prerequisite knowledge/skillset).

Most of these should be common sense to educated, but ensuring that baseline is explicitly mentioned helps set some expectations for the target audience the documentation will be addressing

Kirael12 commented 1 year ago

Hi everyone,

I've started to write some guides for my pool website about monitoring tips. I could help contribute the revamp on the monitoring section. Is there anyone already working on it ?

sanskys commented 1 year ago

I've started to write some guides for my pool website about monitoring tips. I could help contribute the revamp on the monitoring section. Is there anyone already working on it ?

Hi, great that you wish to contribute..welcome. I am working the monitoring topic and would in principal update my current monitoring tutorial - https://developers.cardano.org/docs/stake-pool-course/handbook/grafana-dashboard-tutorial What additions are you planning?

Kirael12 commented 1 year ago

I've started to write some guides for my pool website about monitoring tips. I could help contribute the revamp on the monitoring section. Is there anyone already working on it ?

Hi, great that you wish to contribute..welcome. I am working the monitoring topic and would in principal update my current monitoring tutorial - https://developers.cardano.org/docs/stake-pool-course/handbook/grafana-dashboard-tutorial What additions are you planning?

I was thinking about reverse proxy setup for a secured access to Grafana Monitoring from anywhere (without ssh port forwarding tunnel), GOauth 2FA, hardening tips. I wrote these guides, i would be happy to uodate and add them to your section :)

sanskys commented 1 year ago

sounds good...it might fit even better to the sections - SPO security considerations and Logging and Security

lets wait for other views

rphair commented 1 year ago

@Kirael12 @sanskys it does sound from what the two of you said above that the content could 1) be worked into these sections:

SPO security considerations Logging and Security

... or 2) if it doesn't work well with the existing material, or if @Kirael12 you are taking operators on a different track & want to keep all the material together, we tentatively approved (in discussion beginning here, also with @rdlrt: https://github.com/cardano-foundation/developer-portal/issues/737#issuecomment-1276570746) a section for special topics (Appendix or Deployment Scenarios) where you might post your guide. 🤔

Kirael12 commented 1 year ago

I think it might fit well right after the Grafana setup as it's directly related, and can be easily done right after Grafana install

And then ("advanced") :

But on the other hand, all of this is optional (well, setting up a Grafana Server is not really mandatory either), so it might fit into "Security Considerations" as well :)

rdlrt commented 1 year ago

I feel these are better suited in deployment scenarios too (as neither grafana nor choice of reverse proxy is mandatory, while some might prefer minimalistic approaches) - For instance:

Setting up Nginx Reverse Proxy on Grafana Server Setting up SSL certificate Let's Encrypt Hardening Nginx

One can argue these are easily achieved for someone new using caddy instead - as it avoids set up using certbot/automatic renewal process/etc - an install and conf file is enough and works well with containers/orchestrators , or similarly want to add a different version of 2FA provider (which may also include installation/addition of PAM modules - something that those paranoid with security are uncomfortable with). While in deployment scenarios, these can be easily added/extended with scope for individual bias (that bias is not a bad thing, just natural to all of us - and simply describes how different setups look like).

With modular setup scenarios, you'd be able to add solution that can plug different monitoring frontends, proxy, 2FAs, SSH configs, containerisation, etc - while the heart of entire guide sticks to using CLI/Node without much additional software bias (as much as possible)

rphair commented 1 year ago

@os11k (here) What do you think about adding it here ... a separate guide or page titled "Running a Pool in Docker" or something similar.

I think with everything else said since your comment, an extra page in the flow of the regular SPO course would be welcome. I say this based on a couple years' observation (not my own experience, but on the forum & issue queues) that some SPOs determine at the beginning that the Docker based configuration is going to be best for them... so it would help to have practical advice on how to do this, suitable (like your forum posting) for people doing an initial pool setup.

os11k commented 1 year ago

@os11k (here) What do you think about adding it here ... a separate guide or page titled "Running a Pool in Docker" or something similar.

I think with everything else said since your comment, an extra page in the flow of the regular SPO course would be welcome. I say this based on a couple years' observation (not my own experience, but on the forum & issue queues) that some SPOs determine at the beginning that the Docker based configuration is going to be best for them... so it would help to have practical advice on how to do this, suitable (like your forum posting) for people doing an initial pool setup.

Sounds great! Thank you for your feedback!!! I will add separate page then and no need to wait while Revamp Stake pool is done. :)

Kirael12 commented 1 year ago

I feel these are better suited in deployment scenarios too (as neither grafana nor choice of reverse proxy is mandatory, while some might prefer minimalistic approaches) - For instance:

Setting up Nginx Reverse Proxy on Grafana Server Setting up SSL certificate Let's Encrypt Hardening Nginx

One can argue these are easily achieved for someone new using caddy instead - as it avoids set up using certbot/automatic renewal process/etc - an install and conf file is enough and works well with containers/orchestrators , or similarly want to add a different version of 2FA provider (which may also include installation/addition of PAM modules - something that those paranoid with security are uncomfortable with). While in deployment scenarios, these can be easily added/extended with scope for individual bias (that bias is not a bad thing, just natural to all of us - and simply describes how different setups look like).

With modular setup scenarios, you'd be able to add solution that can plug different monitoring frontends, proxy, 2FAs, SSH configs, containerisation, etc - while the heart of entire guide sticks to using CLI/Node without much additional software bias (as much as possible)

Do you mean moving Grafana Sections to "Deployment Scenarios", and then adding guides i mentionned above there (actually I wrote these guides for a Grafana setup without docker, that is why it's quite straightforward after @sanskys guides :))

rphair commented 1 year ago

OK, @Kirael12 has submitted the material above & I've created a Deployment Scenarios section for it as per the recent comments.

@thenic95 @katomm @rdlrt I've added the new section to the outline here: https://github.com/cardano-foundation/developer-portal/issues/737#issuecomment-1217978885

sanskys commented 1 year ago

@rphair please mark "Monitoring the Node (Prometheus, Grafana, gLiveView)" in the tasks as done. Next i will work on "Generating wallet keys (Faucet for tADA)"

sanskys commented 1 year ago

@lodl how far are you with these section? 

Logging and Security (ssh key,2FA)
User Setup
Server Setup (optimize performance, chrony, updates)
sanskys commented 1 year ago

@weqanhet how far are you with these section?

Rotate KES
Update Registration
Cardano Node Upgrades
rphair commented 1 year ago

@sanskys: mark "Monitoring the Node (Prometheus, Grafana, gLiveView)" in the tasks as done

✔️

sanskys commented 1 year ago

Please tick - Generating wallet keys (Faucet for tADA) next i will work on - Generating Cardano Block producer Keys (StakePool certificate generation)

rphair commented 1 year ago

done & merged #971

btbf commented 1 year ago

About Improve Grafana Security https://github.com/cardano-foundation/developer-portal/blob/revamp-stake-pool-course/docs/stake-pool-guide/improve-grafana-security.md

Very good documentation on improving Grafana security. However, there are a few obstacles to this configuration.

Here's what we know now The HTTP methods are restricted, so some features such as alert rules, data sources, user settings, etc. are not available. (PUT/PATCH/DELETE methods must be allowed)

nginx websocket is not enabled, which causes a large number of websocket errors to be spit out to the browser console, thereby breaking the login status.

I have already found a solution for the above two, but I am continuing to investigate other errors that occur from time to time. I will create a pull request as soon as all solutions are found

rphair commented 1 year ago

thanks @btbf - @Kirael12 what do you think?

Kirael12 commented 1 year ago

Hello @btbf @rphair

We can indeed allow PUT method, especially if you need to modify Alarms, although It works fine without this method allowed, if your configuration is done.

About nginx websocket, as far as i'm concerned, i don't get any errors without it on my server or brower when logging in GRAFANA, but i'm very interested to see what you found on your side :)

(I planed a small pull request to clean the grafana login screen and only show Google OATH method.)

btbf commented 1 year ago

@Kirael12 All right. Yeah,I think you're not wrong.

However,Someone may want to delete and recreate the dashboard. Someone may have set up alerts but want to remove and recreate them. Those actions were feasible, at least in the case of nginx non-passing. Those who set up as per this document will probably end up on the street. Flexibility is necessary. image image

About Websocket They do not occur at login, but appear in the console after login. Are you ignore this? If there is an error, it means that there is some problem and some functions are not available. image

rphair commented 1 year ago

thanks @btbf @Kirael12, great ideas - shall we create an issue in this repo to discuss these related questions, and leave this issue for outline structure & coordinating tasks?

rdlrt commented 1 year ago

In a very small subset of readers, you can already see the variance in usage (which is not surprising - given the discussions over years across mediums) - hence, I very strongly feel these are more fit for deployment scenarios rather than general sections (and then one can pick and customise steps they need, rather then aiming for single use-case attached to things like google auth)

rphair commented 1 year ago

absolutely @rdlrt & no worries since this page is already mounted there (I put it there in https://github.com/cardano-foundation/developer-portal/pull/918):

label: "Deployment Scenarios",
        items: [
        "stake-pool-guide/improve-grafana-security",
                ],
        },

and maybe @btbf @Kirael12 if there are further options for this subset of readers to consider, then there could be more than one article on the subject (perhaps one for each of you, if it's not practical to combine your methods) or even a subdirectory for Grafana configurations. 🤔

Kirael12 commented 1 year ago

Hey @rphair and @btbf

I will make a PR to modify the initial configuration i made.

These are not really options, just corrections, so no need for a subdirectory ;)

os11k commented 1 year ago

Hi! I got today question in telegram about "official" documentation how to claim rewards. Is this something we are planning to add? Do we even need it here? I might try to contribute for this part. What you guys think?

rphair commented 1 year ago

@os11k I think it's a good idea to add to the outline. As I remember (before the Developer Portal hosted this documentation) it was in the instructions that I used (not CoinCashew!) to create my pool in August 2020. From most operators' perspective it is part of the regular workflow of running a stake pool (for those that get blocks).

Personally I considered it vital to document for a customised operating environment as the first of 3 basic stake pool transactions . In one of the most commonly documented & simplest configurations, they will be exposing their pledge address private key to pay the transaction fee for the rewards withdrawal: so it would help to have a run-through on how to do this securely.

sanskys commented 1 year ago

please check Generating Cardano Block producer Keys (StakePool certificate generation)

and next i will work on Registering a Pool (JSON Metadata) Launching Cardano Node (Startup scripts, Systemd)

rphair commented 1 year ago

FYI for those following this issue: I was following two bouncing balls at the same time: whether this new documentation set was supposed to break with using a testnet scenario for building the pool as the previous, soon to be obsolete video documentation & exercises also did... expecting that more of our material would provide cut & paste instructions to build their pools on Mainnet.

As per https://github.com/cardano-foundation/developer-portal/pull/1033#issuecomment-1525548547, apparently we are still advising new SPOs to build their pools first on one of the testnets which I think is debatable, although I'm not going to debate it now & only wanted to update other editors & reviewers because nothing has been commented here in a while. 😎

katomm commented 1 year ago

It's been active for a very long time now, and we've made some good updates. We should think of starting to merge it even if not everything is complete yet.

sanskys commented 1 year ago

It's been active for a very long time now, and we've made some good updates. We should think of starting to merge it even if not everything is complete yet.

Please wait for 2 weeks. I am writing the last chapter of "Cardano Block Producer Configuration", after that it would be ready for merge

katomm commented 1 year ago

@sanskys we can't wait two more weeks, I'm preparing to merge with staging for tomorrow. We will just go with what we have, worth to look into #1057

Once the revamped section is in merged to main we can still iterate to improve it further. Thanks!

sanskys commented 1 year ago

@katomm fine..i would have additionally done some housekeeping (links etc) but then we do it after the merge

katomm commented 1 year ago

Thank you everyone for working on this, please note that with #1063 we are closing this issue. This is not the end of improving the stake pool operator section but was an important step forward. From now own we can just iterate on the smaller changes.