Simple web publishing on 3bot

[rkhamis]

Goal

The idea here is to have any 3bot user able to publish content on their own 3bot This means that the 3bot will be acting as a DNS server in itself, able to register domains. If someone adds this 3bot in their reslov.conf, they'll be able to access these domains.

To achieve this, we'll run a DNS server in a gevent loop

Requirements

• [ ] new endpoints:
  • register_website
  • takes as input:
    • repo URL to publish from (for wiki or website)
    • domain it will publish to
  • the actor will:
    • pull the repo content and generate docs (see similar work done here )
    • check for updates every five minutes (gevent greenlet with sleep can be used here)
    • register a new record in the DNS Server
  • unregister_website
  • takes as input:
    • domain to unregister
  • the actor will:
    • remove the related record from the DNS server
    • remove the repo from the filesystem
• [ ] if DNS cannot resolve the address, it should fallback to 8.8.8.8, to reach another 3bot for instance
• [ ] check if nginx can accept the DNS and forward to the digitalme DNS in the geventloop (nginx is proxy of the DNS)
• [ ] config info must be managed by kosmos config classes
• [ ] automatically publish:
  • in case of website:
  • www is master branch
  • www2 is development branch
• in case of wiki
  • wiki is master branch
  • wiki2 is development branch
• [ ] self-signed certificates for https

Child of threefoldtech/home#162

[rkhamis]

Estimation of work: 5 working days + 3 days for QA

[rkhamis]

➤ Abdulrahman Hamdy commented:

Started exploring the jsx dns server, found out that it resolves the domains using default name server Also trying with nginx virtual hosts with lapis

[rkhamis]

➤ Rana Tarek Hassan commented:

Had a look at 3bot structure and dns server code. Started writing script with schema to save/retrieve to/from bcdb

[rkhamis]

➤ Abdulrahman Hamdy commented:

Finished enabling using of virtual hosts in lapis. Added new branch in lapis-wiki repo to work on these changes called "open_publish"

[rkhamis]

➤ Rana Tarek Hassan commented:

Basic initial schema and script to use save/retrieve to/from db using redis client. Still needs adjustments.

[rkhamis]

➤ Abdulrahman Hamdy commented:

Started creating publishing tool, near to finish the wiki publishing part, will complete it to day

[rkhamis]

➤ Rana Tarek Hassan commented:

• Changed db saving/retrieval to bcdb instead of redis directly.
  • Using this in DNS to resolve the IP by searching in the db before using dns resolver to get an IP

[rkhamis]

➤ serboctor commented:

I helped Rana refactor the bcdb/redis part and make it work. I am currently looking into the dns structure to see if we can return the same structure from our dns resolver or not.

[rkhamis]

➤ Rana Tarek Hassan commented:

• Added new branch in digitalmeX -> development_dns
  • https://github.com/threefoldtech/digitalmeX/commit/9037082e39b52afda400266792895cb7c4ee3c50

[rkhamis]

➤ Rana Tarek Hassan commented:

• Looking into issue where restarting bcdb(using SQLITE) results in retrieval by name of old records to fail. new objects can be saved and retrieved within the same session of running bcdb, but cannot be retrieved it restarted
  • When using zdb as backend db : the model obj created isnt saved to the db

[rkhamis]

➤ serboctor commented:

I worked with Rana on investigating the issues mentioned above.

[rkhamis]

➤ Abdulrahman Hamdy commented:

• Created Gedis actors to deploy wikis and website
  • Configured auto pull for repos using gevent
  • Configured the open_publish client to auto deploy two instances of the wiki and website (development and master)
  • Configured the nginx files to work on both types (wiki/website)

[rkhamis]

➤ Rana Tarek Hassan commented:

• Investigated further with the bcdb issue but no fix reached. Opened a new issue https://github.com/threefoldtech/jumpscaleX/issues/343
  • Changed backend to us zdb instead of SQLITE https://github.com/threefoldtech/digitalmeX/commit/01c218e58f6098dbf7d3c66b542f9994d705e4bf

[rkhamis]

➤ Abdulrahman Hamdy commented:

Managed to test wiki deployment manually on ovh nodes yesterday with Peter and it worked well with two branches, master-development with auto deployment of two subdomains www and www2. Also tested the auto pull and all worked fine. Now trying to create flist to auto start the framework

[rkhamis]

➤ Rana Tarek Hassan commented:

Working on using the web_publishing actor in the chatflow to register/unregister wikis/websites

[rkhamis]

➤ Rana Tarek Hassan commented:

• Worked with @arahmanhamdy on adding starting the required servers to the openpublish client https://github.com/threefoldtech/jumpscaleX/commit/695914d2c644af7f1b0d1dfdced2c7787406864e
  • Added docs for DNS server and DNS record https://github.com/threefoldtech/digitalmeX/commit/35e793a9e3329942a361fbe15424b8931c61e9b5

[rkhamis]

➤ Abdulrahman Hamdy commented:

Refactoring the open publish to be under tools and use it to publish actors and chat flows

KDS asked for some changes in the dns, also writing docs

[rkhamis]

➤ Rana Tarek Hassan commented:

Working on fixing DNSRecord and its schema to have zones with domains for each zone entry

[rkhamis]

➤ Abdulrahman Hamdy commented:

Finished moving open publish into js tools , tested and writing docs

[rkhamis]

➤ Rana Tarek Hassan commented:

Updated DNSResolver to have zones(consist of 2 parts only) https://github.com/threefoldtech/digitalmeX/commit/8c8d1a38c7026d881b6f538b487406f439b245b0

[rkhamis]

➤ Abdulrahman Hamdy commented:

Hardened the adding sites/wikis. Fixed auto reload settings. Tried multiple ways for loading actors (TBD) Completing docs for open_publish