Split ZeroNet into multiple parts

[filips123]

Is your feature request related to a problem? Please describe.

ZeroNet source code is currently pretty huge. It's hard to only use only one part because all parts are very connected with each other.

This is also related to #1289, #2062 and some other comments on GitHub and ZeroTalk.

Describe the solution you'd like

ZeroNet source code should be split into multiple packages. That packages should be for core (communication between peers and related protocol things), user-related things (configuration, saving sites...), WebSocket server (for integration with other software), UI server (for complete ZeroNet like it is today), command-line interface and additional plugins. And maybe also some additional packages if needed.

This would allow easier development of ZeroNet itself and also easier integration with other software.

Also, there should be proper documentation of ZeroNet protocol so it would be easier to make ZeroNet implementations in other languages.

[purplesyringa]

Just warning you that it might take a while. These parts are very integrated into each other.

[wandrien]

Oh, that's complicated...

I guess, it is better to start the refactoring from several easier tasks. For example, moving the BitTorrent tracker related code into the separate plugin, in the same way as the support of zero:// trackers is incapsulated in a single plugin.

[0polar]

Plan:

• Peer discovery / Network connection
• File system / Download management / Big file
• SQL database construction from files
• UI / API / SDK

[0polar]

It is annoying that ZeroNet use non-standard, that make our can not reuse existing tools and neet to reinvent wheels.

For example:

• Custom diff protocol (not Unix)
• WebSocket as RPC (not HTTP)
• Custom protocol for file download and big file (not BitTorrent)

[purplesyringa]

WebSocket as RPC (not HTTP)

What would you use instead? HTTP pooling? It's a lot slower.

[filips123]

@0polar I agree that some things are not standard but this is because there is currently no popular and good standard for such apps.

Custom diff protocol

What do you mean with that?

WebSocket as RPC

Why is this not good? WebSockets are standard. And they are actually connected via HTTP. Other solutions (just HTTP) are slower and not so good for websites that always needs to be connected to ZeroNet instance.

[0polar]

Custom diff protocol

https://zeronet.readthedocs.io/en/latest/help_zeronet/network_protocol/#possible-diff-opcodes

WebSockets are standard.

Half truth. WebSocket is not popular as normal HTTP. HTTP are included in almost every standard library, but WebSocket not.

always needs to be connected to ZeroNet instance

Long connections are stateful, stateful means potential hazard to to concurrency. WebSocket should be opt-in.

[purplesyringa]

Half truth. WebSocket is not popular as normal HTTP. HTTP are included in almost every standard library, but WebSocket not.

Once again. HTTP pooling is a lot slower than WebSocket's.

[0polar]

Argue by repeating is not effective. Read again my comment above.

[filips123]

Half truth. WebSocket is not popular as normal HTTP. HTTP are included in almost every standard library, but WebSocket not.

They are standardized.

I agree that they currently sometimes don't have a library, but this is because they are new. In the (probably near) future, they will be integrated into more things. And don't forget that HTTP also didn't have many libraries when it was invented.

Do you have any special concerns about the availability of WebSocket library? There are WebSocket clients for Node.js, Python, PHP, C, C++, C#/.NET, Rust, Ruby, Go and Java. For even bigger list see Awesome WebSockets.

---

Also, I'm currently creating a WebSocket based ZeroFrame client for local programs in multiple different languages. It will enable usage of ZeroNet features in command programs, desktop apps and classic servers.

Currently, I'm creating JavaScript client (for Node.js and Electron). It's currently available as GitHub Gist, but it will be also available as NPM package (probably) next week. See my ZeroTalk topic for details.

I'm also creating clients for Python and PHP. They will probably be available this month. And I will also try to create clients for other languages soon.

---

Long connections are stateful, stateful means potential hazard to to concurrency. WebSocket should be opt-in.

Do you also have any special concerns here? What would you recommend instead?

[0polar]

If you to remote procedure call from client to server, HTTP is enough. It even can be semantic if REST. You can use WebSocket or raw TCP or whatever to re-implement remote procedure call, but it is not necessary, and it will lost existing tools.

In one word: do not reinvent wheel.

[filips123]

remote procedure call from client to server

But ZeroFrame also calls a client from the server. This is not possible with HTTP or REST.

And I still think that using WebSockets is not reinventing the wheel. WebSockets are also supported in most languages and tools so so you won't lose most existing tools.

[purplesyringa]

This. Websockets are instant, that's the reason why they were invented. The client & the server need to communicate instantly, the messages need to be sent in both directions. HTTP supports neither of this.

[wandrien]

@0polar

Custom protocol for file download and big file (not BitTorrent)

How is it possible to use BitTorrent for dynamic data?

[0polar]

How is it possible to use BitTorrent for dynamic data?

Yes, BitTorrent can't proactively push metadata (content.json), but can be used to normally download files.

[wandrien]

Yes, BitTorrent can't proactively push metadata (content.json), but can be used to normally download files.

Makes sense.

I have an idea to make a plugin to download torrent files into the ZeroNet data folder. The data may be addressable by something like http://127.0.0.1:43110/torrent/<info_hash>/<file_name>.