The problem with your file structure:

[didlie]

This is a huge issue with your file structure, and almost everyone who uses github as their code source: Your dependencies are not in your repository.

Here is the first roadblock reading your code:: I download zeronet.master unzip it, and open in Atom editor. --I enter the program in zeronet.py, and ignore the "import"s knowing they are simply required functions in known thirdparty code blocks... --I find the update section in zeronet.py and comment it out... I don't want my app updating from your github repository. --I move to the main.py file that is initiated in zeronet.py and discover this third party file called gevent that us used to import "monkey" --- i look around in the code base and... no gevent and no monkey.

[skwerlman]

preinstalled 3rd party deps are actually a bad thing, since it makes it difficult for users to update them. as far as getting the deps, you can just pip install -U -R requirements.txt from the root of the repo, same as any other python project. if you're worried about polluting the global package space, just use a venv

[HelloZeroNet]

Gevent and msgpack are binary modules (written in C), so you need to compile them before you can use it, so it's not possible to distribute it with the source code.

[ghost]

I actually completely agree with @didlie that dependencies should be in github repo. Also something to note, this does not mean that these deps cannot be updated or that you can't use pip (or some other package manager). It should be the responsibility of the programmer to update the dependencies in the repo to make sure that the new versions actually work. By storing your dependencies in the repo, this allows people to still use the code even when the places where the dependencies are usually downloaded from are down or no longer exist (cough leftpad cough).

Regarding not being able to store binary files with source code, that wouldn't be as much of a problem if we would quit using Git and use a better system, although one needs to be created first I guess, and it could reduce visibility/discoverability of the project. However, there is LFS.

[purplesyringa]

@krixano That could be some repository with submodules, right? I.e. add all the modules as links to GitHub.

[ghost]

@imachug No, because those repos could be deleted/down and I don't think that git keeps the files you downloaded from the submodules in with your repo while still letting you update them, if it did that, then that would be better in my opinion.

[didlie]

Yes... all dependencies should have direct references

[purplesyringa]

But at least we shouldn't clutter the main repo, so maybe we can fork the modules?

[ghost]

@imachug I don't think forking the modules is that much different, but it's better. Imo, the dependencies should be directly in the same folder so that if you decide to move the repo someone else, you'll still have all of it.

Basically, what I think would be the best is if there was a package manager (like npm) that stores the dependencies in a dependencies folder, and then you just track that folder in with your repo. The problem with doing this with web development (e.g. npm) is that the web has so many dependencies that it would definitely clutter up the directory and may take quite a lot of space. That's also why I think the whole climate of web development isn't very good (I mean come on, people are depending on libraries that literally have barely any lines of code - again leftpad).

[didlie]

Unlocked I archived dependencies waste time for new developers and compromise the existence of valuable projects. When projects go corporate they are destroyed by corrupting dependencies and nieve new developers waste thousands of hours searching for things they can never fix

[ghost]

Thinking about leftpad some more and looking at it's examples and it's source code - I find it even more stupid. Literally all it does is put spaces (or a character of choice) at beginning to make the size of the string the size that you want. The implementation looks even more stupid. They're doing some weird thing in a loop where they divide by two when all they have to do is take the length of the string minus the desired length and add that many spaces to the beginning. It makes no sense to me. It looks they are trying to do some optimization, but is that really needed for something like this?

And this is a library that so many people were using - which is even more ridiculous imo.

Okay, I'm done with that now.

[didlie]

Please put a complete list of all dependencies with links to every dependency for developers to download and create their own archives.

[didlie]

gevent and msgpack are python packages import gevent import msgpack

--- what happens when these python imports import different versions of the same libraries? When I just installed Python, it gave me 3.7, that is not compatible with this repo

[HelloZeroNet]

Any version after gevent 1.0 should work on Python 2.7.x.

All non-binary dependencies are included under the lib directory. For binary we would require to include many different versions based on operating system, cpu type and python version.

[ghost]

When I just installed Python, it gave me 3.7, that is not compatible with this repo

Linux distros separate python2 and python3. Some of them have python default to python2 and some have python default to python3. But you can find both in each of their respective package managers. Not sure if you are on Linux or not though.

[didlie]

windows defaults to python 3

[ghost]

You can install python2 or python3 on windows. Right on the download page they have an option for python2. https://www.python.org/downloads/

If, however, you are talking about the "python" command defaulting to python3 when you have both installed - you can try using the command "python2", but I doubt that will work (that's how it's done on linux, not sure about Windows). But otherwise, I don't know about this.

[skwerlman]

I actually completely agree with @didlie that dependencies should be in github repo. Also something to note, this does not mean that these deps cannot be updated or that you can't use pip (or some other package manager). It should be the responsibility of the programmer to update the dependencies in the repo to make sure that the new versions actually work. By storing your dependencies in the repo, this allows people to still use the code even when the places where the dependencies are usually downloaded from are down or no longer exist (cough leftpad cough).

you can specify known working versions of modules in the requirements.txt

Basically, what I think would be the best is if there was a package manager (like npm) that stores the dependencies in a dependencies folder, and then you just track that folder in with your repo. The problem with doing this with web development (e.g. npm) is that the web has so many dependencies that it would definitely clutter up the directory and may take quite a lot of space. That's also why I think the whole climate of web development isn't very good (I mean come on, people are depending on libraries that literally have barely any lines of code - again leftpad).

this can be solved by using a venv and tracking that. python libs are typically smaller than npm libs, so this is not as big of an issue.

[didlie]

Python and Javascript both need to agree on a system of standardized functions, rather than dependencies and poorly documented as such... with each author / group so proud of their splatter of code that they have their own websites, and the like. PHP's function system, hidden classes, and unified documentation make for a more stable language, as an example to the contrary.

[ghost]

@skwerlman Did you read everything I wrote? I'm saying storing working versions in requirements.txt is not enough because the main sources that you can get these from could go down or be deleted, like with the case of leftpad.

@didlie If you are talking about ZeroNet What is there that isn't standardized or not documented? It's documented that it uses Python2 and it's documented how to run this code for many Linux distros, there's documentation on every API function for ZeroFrame (which takes care of Python and Javascript both need to agree on a system of standardized functions).

Python dependencies are recorded in requirements.txt, which has been said a couple times now. JavaScript dependencies are dependent on what each zite does (aside from pretty much every single zite using ZeroFrame API). Zites are exactly the same as regular sites on the internet, they just replace their communication with a central server for communication with the ZeroNet server/client (by using the ZeroFrame API, some of which replaces browser api functions). That's essentially it.

Here you go:

• Running, Installing, Creating a zite using command line - https://github.com/HelloZeroNet/ZeroNet/blob/master/README.md
• A little bit of how ZeroNet works - https://docs.google.com/presentation/d/1_2qK1IuOKJ51pgBvllZ9Yu7Au2l551t3XBgyTSvilew/pub?start=false&loop=false&delayms=3000#slide=id.g9a1cce9ee_0_4
• ZeroNet documentation (including on ZeroFrame, content.json, and dbschema/databases) - https://docs.google.com/presentation/d/1_2qK1IuOKJ51pgBvllZ9Yu7Au2l551t3XBgyTSvilew/pub?start=false&loop=false&delayms=3000#slide=id.g9a1cce9ee_0_4 (This can also be found as a zite on ZeroNet, the zite being called ZeroNet Docs)
• ZeroNet's python dependencies for the actual application - https://github.com/HelloZeroNet/ZeroNet/blob/master/requirements.txt
• You can find a tutorial for creating a zite with a database on ZeroBlog.
• I believe imachug has already mentioned his tutorials to you.
• There is one tutorial on creating a ZeroNet Plugin on my ZeroNet Dev Center
• I also have tutorials on how ZeroNet works overall and a little of why it works the way it does - all on ZeroNet Dev Center.

The documentation is there for the majority of stuff for ZeroNet, it just needs vastly improved.

If you are talking about Javascript and Python themselves:

• https://docs.python.org/3/
• https://docs.python.org/2.7/
• MDN (Mozilla Development Network) provides documentation on pretty much everything in Javascript, including which browsers implement what
• The standard of Javascript is called ECMAScript: http://www.ecma-international.org/publications/standards/Ecma-262.htm
• https://www.w3.org/standards/ and https://www.w3.org/standards/webdesign/script - define the browser APIs that have been developed by the W3C

Saying that python and/or Javascript don't have a set of standardized functions and no documentation is wrong, clearly. Furthermore, if you are saying that they both need to have the same set of standardized functions, I completely disagree. If you are saying that they both each need to have standardized functions of their own, then I believe, as shown above, that they do.

Although, I do agree that web developers need to learn how to write code themselves instead of just using libraries for every single thing. However, I think using stuff like Vue, is fine because it would take forever to create those things. The only other library that may be useful is an editor library because the standard for contentEditable is horrible and it's implementations are buggy and different with each browser.

Btw, PHP is a horrible language. So is Javascript. Python I don't like much. The only decent programming languages that I've used/seen I would say are C, Rust, JAI (unreleased), and Odin.

[skwerlman]

Did you read everything I wrote? I'm saying storing working versions in requirements.txt is not enough because the main sources that you can get these from could go down or be deleted, like with the case of leftpad.

Right, which is what the venv is for. You still have the deps in the repo, but they are directly upgradeable by standard dependency management tools, a direct improvement over the current situation. In case what I mean by venv isn't clear, docs are here:

• PY3's venv module: https://docs.python.org/3/library/venv.html
• the virtualenv tool for PY2: https://virtualenv.pypa.io/en/stable/

I think a longer term solution could be to mirror and fetch the needed deps on some decentralized file system, such as IPFS or a zite, but that's probably out of scope for zeronet

Btw, PHP is a horrible language. So is Javascript. Python I don't like much. The only decent programming languages that I've used/seen I would say are C, Rust, JAI (unreleased), and Odin.

I feel like this is a poor battlefield for the language wars. (although, ye, php and js are horrible)

[ghost]

Is virtualenv like what Docker does? Sorry, I don't know much about that stuff. But if it is, it's kinda dumb because it solves a problem that wouldn't have been a problem if people would just quit using system libraries.

[purplesyringa]

@krixano If you used Wine, virtualenv is something like prefixes in Wine.

[didlie]

@krixano really vague documentation, but thank you anyway.

re:leftpad... python is not a stable language.

dependencies should be included in the repo....

I want a full blueprint / logic flow of the zeronet/zeronet - from the initial download, install and identification of peers and self-identification and creation of the welcome page... minimalist

[ghost]

I want a full blueprint / logic flow of the zeronet/zeronet - from the initial download, install and identification of peers and self-identification and creation of the welcome page... minimalist

Not sure precisely what you are talking about here, but the powerpoint/slides I linked to gives some information on how a zite is downloaded (this includes the ZeroHello "welcome" page, it is just a zite that is downloaded), checked for authenticity (using signatures), etc. My tutorials include some of this as well. My tutorials also goes over databases and users and a little of how ZeroId and KaffieId work. I would say my tutorials aren't just for ZeroNet Zite development, but also to get a little bit of an understanding of how ZeroNet works overall. I would reccommend my first tutorial "The Basics", and my "Users and Databases In ZeroNet" tutorial. If you want to go further with actual zite development, then @imachug 's tutorials are good - even if you just want to learn a little of how merger zites and optional files work. The only other thing I can suggest is look at example websites - many of them are on GitHub (and GitCenter), including the "official" ones (ZeroBlog, ZeroId, ZeroMe, etc.). Otherwise, I don't know much about the actual internals of ZeroNet (besides what I've provided in my tutorials, which really just gives a simple overall view).

The documentation can be a little vague sometimes and some parts of it rely too much on examples (and not enough on descriptions/specifications) I would say. Although, I think the best documented part is the ZeroFrame API commands.

[didlie]

@krixano

REF: https://github.com/karpathy/convnetjs/blob/master/build/deepqlearn.js

Example of code commenting:

(function(global) {
  "use strict";

  // An agent is in state0 and does action0
  // environment then assigns reward0 and provides new state, state1
  // Experience nodes store all this information, which is used in the
  // Q-learning update step
  var Experience = function(state0, action0, reward0, state1) {
    this.state0 = state0;
    this.action0 = action0;
    this.reward0 = reward0;
    this.state1 = state1;
  }

  // A Brain object does all the magic.
  // over time it receives some inputs and some rewards
  // and its job is to set the outputs to maximize the expected reward
  var Brain = function(num_states, num_actions, opt) {
    var opt = opt || {};
    // in number of time steps, of temporal memory
    // the ACTUAL input to the net will be (x,a) temporal_window times, and followed by current x
    // so to have no information from previous time step going into value function, set to 0.
    this.temporal_window = typeof opt.temporal_window !== 'undefined' ? opt.temporal_window : 1; 
    // size of experience replay memory
    this.experience_size = typeof opt.experience_size !== 'undefined' ? opt.experience_size : 30000;
    // number of examples in experience replay memory before we begin learning
    this.start_learn_threshold = typeof opt.start_learn_threshold !== 'undefined'? opt.start_learn_threshold : Math.floor(Math.min(this.experience_size*0.1, 1000)); 
    // gamma is a crucial parameter that controls how much plan-ahead the agent does. In [0,1]
    this.gamma = typeof opt.gamma !== 'undefined' ? opt.gamma : 0.8;

    // number of steps we will learn for
    this.learning_steps_total = typeof opt.learning_steps_total !== 'undefined' ? opt.learning_steps_total : 100000;
    // how many steps of the above to perform only random actions (in the beginning)?
    this.learning_steps_burnin = typeof opt.learning_steps_burnin !== 'undefined' ? opt.learning_steps_burnin : 3000;
    // what epsilon value do we bottom out on? 0.0 => purely deterministic policy at end
    this.epsilon_min = typeof opt.epsilon_min !== 'undefined' ? opt.epsilon_min : 0.05;
    // what epsilon to use at test time? (i.e. when learning is disabled)
    this.epsilon_test_time = typeof opt.epsilon_test_time !== 'undefined' ? opt.epsilon_test_time : 0.01;