Open jsmerel opened 9 years ago
In going through tutorials on the new documentation for Thunder 1.0.0 (docs.thunder-project.org), I noticed a few similar issues that could be confusing. They are mostly things that existed from the text of the earlier documentation that didn't get switched to match the code for the new 1.0.0 structure. Following the code is always correct and works as expected, but some of the discussion between code blocks wasn't changed appropriately. Here are a few I noticed:
Again, following the code is clear enough to understand what is happening, but it might be confusing for some individuals coming from matlab and not as familiar with Python. I noticed some of these things didn't change and recognized the old versions from the previous tutorials, so I thought I'd document it here for the next time somebody updates the documentation.
Hey, I started trying this out and went through this tutorial: http://thunder-project.org/thunder/docs/install_ec2.html
There were a few dumb nuances which confused me slightly (as a total novice). I figured I'd state these explicitly in case there is any desire to clarify them in the documentation. Note: Jeremy was sitting next to me so these were resolved quickly...
1) In the opening section "Setting up an Amazon account", bash_profile also goes by .bashrc (e.g. on ubuntu). For emphasis, this could be stated explicitly to be on the local computer, prior to login to the cluster.
2) It is mentioned when talking about setting up the ipython notebook, "If this is the first time you are logging in, you must run a script that configures iPython notebook to run in SSL mode." The nuance is that after a stop and start this needs to be done again. In retrospect this is obvious because a new cluster is instantiated, but to me as a novice this was slightly non-obvious.
3) Along with the ipython notebook details, a mention of the availability of http://.....amazonaws.com:8080 would be helpful. A trick such as 8080 is admittedly not just a thunder thing -- perhaps just a link to documentation on this would be useful. Could also mention the issue of if RAM is full, then object spills to disk (or maybe this is not thunder specific enough).
3) A presumably temporary problem is that before running the ipython notebook, one needs to run pip install jupyter. Jeremy suggested this will be solved at some point. But for novices it might be mentioned how pip install works on master/slave nodes. Jeremy sent me: pssh -h /root/spark-ec2/slaves 'source ~/.bash_profile && pip install bolt-python' when I needed to install bolt on slaves. Had he not been here, the nuances of this would have been non-obvious. This could be documented for other uses who need packages. If this is outside the scope of the thunder documentation, a relevant link would at least be useful.
Also, some slightly advanced documentation might be useful. For example, I would benefit from a brief layout of a few things like ec2.py being the script that controls the installations on thunder -- maybe this already exists somewhere or is considered irrelevant, but a few paragraph description of the layout structure of the source might make it more accessible for developers (or maybe they're all smart enough to figure it out without this).