zerobot packages cleanup [development_actorsimprovement]

despiegk commented 4 years ago

checklist

[x] there is not enough structure in the actors, they need to be reviewed and cleaned up ~~- [ ] we need example how we will do the solution packages for tfgrid e.g. wordpress, ...~~ will be addressed in tf solution related story
[x] documentation needs to be part of packages as a wiki subfolder
[x] lets create quite some documentation for existing packages
[x] need new decorators on all actor methods (see apps package)
[ ] need to make sure we only use j.threebot... (no longer the self.... on packages)
[x] make sure factory classes only use threebotserver (not in own process)

rkhamis commented 4 years ago

To review + document:

package factory should j.baseclasses.threebot_factory
no manual loading of actors or models
no manual loading of wikis
move relevant package docs into wiki dir in the package
- [x] demo
  - [x] fruum @Hamdy
  - [x] pastebin @xmonader
- [ ] examples
  - [ ] helloworld
  - [ ] ibiza
  - [x] vuejs @xmonader
- [x] tfgrid -- review only @ranatrk and @alichaddad
  - [x] phonebook
  - [x] registry
  - [x] tfgrid_directory
  - [x] tfgrid_dns https://github.com/threefoldtech/jumpscaleX_threebot/issues/176
  - [x] tfgrid_network
  - [x] tfgrid_token
  - [x] tfgrid_workloads
  - [x] registration
- [ ] threefold
  - [x] communityapp @BolaNasr
  - [ ] phonebook -- review @ranatrk (duplicate to one in tfgrid)
  - [x] tfgrid_directory @xmonader
  - [ ] tft_explorer @zgorizzo69
  - [x] threebot_provisioning @ashraffouda
  - [x] threefold_websites @alichaddad
  - [x] threefold_wikis @abom
- [x] zerobot
  - [x] alerta @xmonader
  - [x] myjobs @xmonader
  - [x] packages_manager @xmonader
  - [x] webhooks @ranatrk
  - [x] webinterface @andrewayoub
  - [x] wiki @andrewayoub
[x] tool to bootstrap new packages @xmonader

despiegk commented 4 years ago

probably need to merge [development_actors_improvement]

abom commented 4 years ago

Fixed this gedis cmd and actor_method errors at https://github.com/threefoldtech/jumpscaleX_core/pull/284/commits/9b8256900efc6b0cc699f58ddaeabfe33c69cdca

When threebot server start:

Thu 21 11:40:03 s/gedis/GedisCmds.py - 186 - _method_source_process             : EXCEPTION:                                        

--TRACEBACK------------------                                                                                                       
/sandbox/lib/jumpscale/Jumpscale/servers/threebot/ThreebotServer.py in start                                                        
    286    package.start()                                                                                                          
/sandbox/lib/jumpscale/Jumpscale/tools/threebot_package/ThreeBotPackage.py in start                                                 
    124    self._init_before_action()                                                                                               
/sandbox/lib/jumpscale/Jumpscale/tools/threebot_package/ThreeBotPackage.py in _init_before_action                                   
    64    self.gedis_server.actors_add(path, namespace=self._package_author.actors_namespace)                                       
/sandbox/lib/jumpscale/Jumpscale/servers/gedis/GedisServer.py in actors_add                                                         
    143    self.actor_add(file_path, namespace=namespace)                                                                           
/sandbox/lib/jumpscale/Jumpscale/servers/gedis/GedisServer.py in actor_add                                                          
    164    self.cmds_meta[key] = GedisCmds(server=self, path=path, name=name, namespace=namespace)                                  
/sandbox/lib/jumpscale/Jumpscale/servers/gedis/GedisCmds.py in __init__                                                             
    87    self._method_source_process(cmd, code)                                                                                    
/sandbox/lib/jumpscale/Jumpscale/servers/gedis/GedisCmds.py in _method_source_process                                               
    186    raise j.exceptions.Base()                                                                                                
-----------------------------

When calling actor_method decorated methods

Thu 21 11:43:20 LASSES/Decorators.py -  10 - process_doc_str                    : EXCEPTION: 
    AttributeError("'NoneType' object has no attribute 'split'",)
--TRACEBACK------------------
<stdin> in <module>
    1    
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/Decorators.py in wrapper_action
    43    self._schemas[name] = process_doc_str(func)
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/Decorators.py in process_doc_str
    10    for line in func.__doc__.split("\n"):
-----------------------------

'NoneType' object has no attribute 'split'

Now me and @ranatrk are trying to update all actors to use this new method and make sure it works.

abom commented 4 years ago

We updated most of actors in https://github.com/threefoldtech/jumpscaleX_core/pull/284 and https://github.com/threefoldtech/jumpscaleX_threebot/pull/272.

Actors work with a client normally, but they don't set user_session parameter when used via j.threebot.actors, so errors like this happen:

JSX> j.threebot.actors.default.package_manager.package_add(path='/sandbox/code/github/threefoldtech/jumpscaleX_thr
   1 eebot/ThreeBotPackages/demo/fruum')
Thu 21 14:53:40 s/package_manager.py -  26 - package_add                        : EXCEPTION:
    AttributeError("'NoneType' object has no attribute 'admin_check'",)
--TRACEBACK------------------
<stdin> in <module>
    1
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/Decorators.py in wrapper_action
    56    res = func(self, **kwargs)
/sandbox/lib/jumpscale/Jumpscale/servers/threebot/base_actors/package_manager.py in package_add
    26    user_session.admin_check()  # means will give error when not an admin user
-----------------------------

'NoneType' object has no attribute 'admin_check'

abom commented 4 years ago

Also, removing builders from j object affected e.g. fruum pacakge

JSX> cl.actors.package_manager.package_add('/sandbox/code/github/threefoldtech/jumpscaleX_threebot/ThreeBotPackage
   1 s/demo/fruum')
default__package_manager
b"Could not add package fruum: 'Jumpscale' object has no attribute 'builders'"

The thing is, like this case, if a package needs to install something via pip or apt how this can be done?

xmonader commented 4 years ago

@Hamdy can you please update your package ?

abom commented 4 years ago

We updated all of actors to use actor_method decorator, but still, the problem of user_session mentioned earlier will prevent calling e.g. pacakge_manager from j.threebot.actors.default.package_mangger...

despiegk commented 4 years ago

create toml files for each package

despiegk commented 4 years ago


[source]
name = "threebot_provisioning"
description = "allow anyone to deploy threebots on TFGRID installed on TFGRID explorers"
threebot = "threefold"
version = "1.0.0"

[actor]
namespace = "threebot"

[[bcdb]]
namespace = "threebot_provisioning"
type = "zdb"
instance = "default"

abom commented 4 years ago

Regarding user_session, @Hamdy told me that we should use threebot.me information when doing calls directly from j.threebot.actors, but his work on this is not yet merged.

Another note for actors in package.toml, we can specify actor namespace, this will break related components like gedis http where we use default namespace, so this related components need to updated accordingly.

abom commented 4 years ago

Need more clarification on these two points

need to make sure we only use j.threebot... (no longer the self.... on packages)

make sure factory classes only use threebotserver (not in own process)

ranatrk commented 4 years ago

The toml files are added in the packages

xmonader commented 4 years ago

blocked on https://github.com/threefoldtech/home/issues/433

xmonader commented 4 years ago

https://github.com/threefoldtech/home/issues/433 is solved and 3bot starts again

I renamed wiki package to wiki_web because wiki path was getting ignored
fixed templates loading in wiki_web package
added more descriptions in tomls

despiegk commented 4 years ago

Regarding user_session, @Hamdy told me that we should use threebot.me information when doing calls directly from j.threebot.actors, but his work on this is not yet merged.

Another note for actors in package.toml, we can specify actor namespace, this will break related components like gedis http where we use default namespace, so this related components need to updated accordingly.

what do you mean with calls from threebot actors? Without TCP? indeed then use the info from threebot.me, please merge ONLY this work in, its needed
yes indeed, we need to specify the namespaces properly and this will brake behaviour, I will now go over the packages and make them right, the toml files

ranatrk commented 4 years ago

Was blocked on threebot server not starting
Worked on making chat work (food_get) : https://github.com/threefoldtech/jumpscaleX_threebot/commit/38232caa24313fc33ebf6d620c383c863e249f3c

Issue: when adding any package, the following error is triggered. Looking into it Screenshot from 2019-11-26 17-50-41

abom commented 4 years ago

Fixed myjobs and wikis loading at https://github.com/threefoldtech/jumpscaleX_core/pull/284/commits/ff9e1bff8eb3e264194d527736a9004f7ef8c299.

Now there're three problems: 1- Sonic indexing is broken

JSX> j.clients.sonic.bcdb.push("a", "b", "c", "t")
Tue 26 17:07:44 ages/sonic/client.py - 117 - _parse_protocol_version            : EXCEPTIO
    ValueError("ENDED authentication_failed doesn't contain protocol(NUMBER)",)
--TRACEBACK------------------
<stdin> in <module>
    1    
/usr/local/lib/python3.6/dist-packages/sonic/client.py in push
    536    return self._execute_command("PUSH", collection, bucket, object, text, lang)
/usr/local/lib/python3.6/dist-packages/sonic/client.py in _execute_command
    465    active = self.get_active_connection()
/usr/local/lib/python3.6/dist-packages/sonic/client.py in get_active_connection
    452    active = self.pool.get_connection()
/usr/local/lib/python3.6/dist-packages/sonic/client.py in get_connection
    375    conn = self._make_connection()
/usr/local/lib/python3.6/dist-packages/sonic/client.py in _make_connection
    397    con.connect()
/usr/local/lib/python3.6/dist-packages/sonic/client.py in connect
    219    self.protocol = _parse_protocol_version(resp)
/usr/local/lib/python3.6/dist-packages/sonic/client.py in _parse_protocol_version
    117    raise ValueError("{} doesn't contain protocol(NUMBER)".format(text))
-----------------------------

ENDED authentication_failed doesn't contain protocol(NUMBER)
JSX>

2- Using dot-separated names as wiki name broke the frontend of wiki package (docsify): Screenshot from 2019-11-26 19-10-18

3- Package loading using the usual way of package manager actor is broken as @ranatrk mentioned earlier.

cl = j.clients.gedis.get(name="threebot", port=8901, namespace="default")
cl.actors.package_manager.package_add(path='/sandbox/code/github/threefoldtech/jumpscaleX_threebot/ThreeBotPackages/zerobot/alerta_ui')

xmonader commented 4 years ago

for sonic problem I'm sure there's a hardcoded value to start the server 123456 used for sonic_config_default, sonic_config_bcdb. It's set correctly in sonic_config_threebot we need to make sure it's starting with that one.
the actors should be called from the threebot shell and there's also j.threebot.package.actors_add that doesn't work?

abom commented 4 years ago

Sonic problem mentioned here is fixed by https://github.com/threefoldtech/jumpscaleX_core/pull/284/commits/fb2b4f25898b6bf5bdfa988d5a3e2b0fc2ac1500, as @xmonader said, it was the wrong auth_password.

abom commented 4 years ago

Now adding packages using actors should work, also from threebot shell (need to pull jumpscaleX_core):

j.threebot.actors.default.package_manager.package_add(path='/sandbox/code/github/threefoldtech/jumpscaleX_threebot/ThreeBotPackages/demo/pastebin')

abom commented 4 years ago

Fixed loading of wikis with dot-separated names at https://github.com/threefoldtech/jumpscaleX_threebot/pull/272/commits/1fd0b4b51b0523fb753354b70a41916c25141426.

The problem was the name of the main div where docsify load wiki content, it need to be without dots and it's not necessary to reflect the name of current wiki.

abom commented 4 years ago

All of issues mentioned here are fixed now.

andrewayoub commented 4 years ago

wikis running on the new convetions for locations /threefold/pkgname/wiki
chat running on the new convetions for locations /threefold/pkgname/chat
there're problems loading the package with package manager actor (still investigating)
update webinterface endpoint to the new conventions
actors endpoint aren't updated yet
crud actors are generated on loading time per package

abom commented 4 years ago

Updated gedis handler to accept new convention here https://github.com/threefoldtech/jumpscaleX_threebot/pull/272/commits/393d521fa5a60931e7f5f076624e5247c503a690 (without breaking current one).

Also, updated package_manager actor to add packages with the new naming convention of <threebot_name>.<package_name> here https://github.com/threefoldtech/jumpscaleX_core/pull/284/commits/7d239c01d2e9df6c0efa36482b3dafcfd6a4c821.

Also, added package gedis client here https://github.com/threefoldtech/jumpscaleX_weblibs/commit/513c7a0dc36321d817a8f3b0a50e9d2f38a9ab70, example usage:

packageGedisClient.threefold.alerta_ui.actors.alerta.list_alerts().then(d => d.json())

abom commented 4 years ago

Now we handle all requests on / using our bottle app https://github.com/threefoldtech/jumpscaleX_threebot/pull/272/commits/c43a788f50ecce04cdbbe2ab9e133551b2748a22#diff-b1756cefba2956256801dc4dbe61a9bfL29, i understand this is needed for new conventions, but how will we support package locations defined in package.py?

abom commented 4 years ago

Now locations are registered using new naming conventions <threebot_name>/<package_name>.

All of packages under zerobot are working (webinterface, myjobs...etc).

ranatrk commented 4 years ago

Updated packagemanager location in frontend to use zerobot/packagemanager instead of threebot/packagemanager and rebuilt frontend to reflect these changes. https://github.com/threefoldtech/jumpscaleX_threebot/commit/939c96c3f840f6ad5785a4af347ea02094b733a9 https://github.com/threefoldtech/jumpscaleX_threebot/commit/8f45f9eec65d592f4ed88909bc780a6c0f6a3c62

xmonader commented 4 years ago

All of zerobot packages are working
Chat package works with the http client now instead of websockets
No more factories in our packages
Alerta_UI is registered by default
Package.toml is properly reviewed by @despiegk
Packages outside of zerobot been skimmed over and renamed @despiegk
Some of them still have factories for tests.

It's better to close this issue now

Dinaamagdy commented 4 years ago

Docs

package.toml missing here https://github.com/threefoldtech/jumpscaleX_threebot#creating-a-new-package
wrong url in almost pacakge ex:myjobs https://github.com/threefoldtech/jumpscaleX_threebot/blob/development/ThreeBotPackages/zerobot/myjobs_ui/wiki/README.md#running instead of https://172.17.0.2/myjobs should be https://172.17.0.2/zerobot/myjobs_ui same for alerta and package manager ,etc -- should add that alerta added by default too here https://github.com/threefoldtech/jumpscaleX_threebot#starting-the-server

xmonader commented 4 years ago

continued here for threebot, tfgrid packages https://github.com/threefoldtech/home/issues/455

abom commented 4 years ago

@Dinaamagdy Done in https://github.com/threefoldtech/jumpscaleX_threebot/pull/301 and https://github.com/threefoldtech/jumpscaleX_core/pull/313.

Dinaamagdy commented 4 years ago

verified on development branch , check that docs compatible with new changes

threefoldtech / home

zerobot packages cleanup [development_actorsimprovement] #377

checklist

Docs