Refactor

[ytti]

Reminder for myself, what things we need to fix when/if refactoring the core.

1. Move state from memory to sqlite. Only keep in memory being fetched right now. So that we scale as number of threads, instead as number of devices
2. Allow models to use multiple input methods, you may want to use ssh to send config, and ftp to actually receive it
3. Extend support of creating multiple files from models, current use-case only conveniently supports creating files in new directories or in new repos. But for stacks, virtual firewalls etc. you might want to create multiple normal files
4. Support multiple source files (multiple times same type, and/or combination of different types)
5. Support multiple outputs
6. Separate device interaction to own library, make Oxidized client of the library
7. Flexible naming of files directory/%g/%t/%n:%p, replacement with group, type, node, port, model, command...
8. Nested groups (unsure) /country/pop ... /customer/x
9. one off run of all devices or just one devices, even while running daemon as well
   10. more hooks, for example when single rotation has been done
   11. testing (eeh, at least for refactored code)

[MajesticFalcon]

Working on #6 now

[ytti]

12. we should support routers pushing config to us, http, ftp, tftp.. listener thread, with worker fetching configs from listener threads via Queue
13. API should be changed to Swagger and return only JSON, with HTTP rendering at client side code + 0touch LE integration
14. review issues to see what users are missing :/

[ytti]

with https://github.com/mitchellh/go-mruby we could rewrite oxidized in golang, but continue supporting ruby device models. Ship everything in single binary, but still support loading local ruby models.

[aakso]

I've been toying with the possible implementation of a go version of Oxidized and so far it seems to be a viable option. The models can be indeed left as ruby code and use the go-mruby to evaluate them.

Some benefits regarding the go-rewrite:

• easier deployment for users. Just download the binary, generate configuration and you are off
• easier memory management: go provides fantastic tools to debug memory utilization
• better isolation: each invocation of model can be run in a separate mruby-instance. Thus all the memory and side effects will be removed once the model has done its work
• rewriting the base gives us an opportunity to fix some of the problems such as test coverage. Testing could be implemented for models too.
• it would be probably a lot faster
• users could still load external models via configuration as before. Builtin models would be packaged into the main binary

And the downsides:

• achieving complete feature parity will take some time
• keeping the original ruby-api in models introduces some constraints but for now it seems to be just a cosmetic issue
• lack of some ruby-libraries for mruby. Though looking at the current state of models this might be a non-issue
• extending builtin models can be a tricky subject. Not sure how much this is used.

I'm planning to release a PoC codebase in coming month(s) or so. So far I have the model evaluator and some basic Input modules done.

[sts]

Deployment could be fixed by shipping as Docker container or providing classical pre-build packages (eg. https://github.com/jordansissel/fpm)?

[ytti]

15. as we separate device interaction from configuration backups we should further develop another software which collects formal set of normalised data from supported devices with motivation of having programmatic platform agnostic access to the data