Ubuntu installation problems

funderburkjim commented 4 years ago

I tried installing on the DigitalOcean development server. But ran into problems that I currently don't know how to resolve.

Problems with git clone

I made a directory /var/www/html/cologne1 cd'd into it, then git clone <csl-pywork>. I think I had to use 'sudo'. It looked like it worked, but when I did a git status, it showed all files 'in red' -- as if all the files in csl-pywork needed to be added and committed. This is different behavior from same 'git clone' on Cologne server, and on local windows Git Bash terminal.

Problem with bash.

In the generate_dict.sh script, there was an error from line dict=${1^^} -- You can make a temporary script to see the error message -- it was something like 'error in expansion'. The shell at DO IS bash (echo $0 -> -bash, which is also the shell at Cologne and of course with Git Bash.

Problem with permissions

I'm not sure what the permissions need to be, and when/how they should be set. When running the installer, I needed to use 'sudo' (sudo sh generate_dict.sh xxx yyy) . I've been told that chmod -R 0777 is bad from a security point of view. But am not sure how things should be set up in /var/www/html.

funderburkjim commented 4 years ago

Permissions solution

The following seems to work.
Assume the login is USER (this isn't a real login name -- use the one you know, if you know).

cd /var/www/html
# use sudo to create a subdirectory of /var/www/html
  sudo mkdir DIR   # DIR is some directory name, such as 'cologne'
# The owner and group will be 'root'
ls -l
# drwxr-xr-x 2 root root  4096 Oct  2 19:59 DIR
# change owner to login user:
sudo chown USER DIR
ls -l
drwxr-xr-x 2 USER root  4096 Oct  2 19:59 DIR
#  To remove DIR and all its contents, sudo required
sudo rm -r DIR
# The good news:  user USER can do most anything WITHIN DIR, 
# and without the need for 'sudo'.
# Normal files and directories show with user=USER and group=USER.
# Normal files seem to have permissions rw-rw-r--  
# and directories have permissions rwxrwxr-x
# Note:  PHP files and html files work properly from browser.
# To see the groups that 'USER' belong to are:
groups USER

funderburkjim commented 4 years ago

Git cloning solution

Assume we are logged in as USER and are in DIR (which is created as above) or some subdirectory of DIR.

To clone a GitHub repository, use the 'HTTPS' method. For example, to clone csl-pywork:

git clone https://github.com/sanskrit-lexicon/csl-pywork.git

This will create subdirectory csl-pywork. For example, if the current directory is DIR, then the git clone command will create DIR/csl-pywork.

funderburkjim commented 4 years ago

Bash parameter expansion solution

Please see: Bash parameter expansion new solution comment below for a better solution

This is because of a difference in the 'sh' command in Ubuntu. /bin/sh is a symbolic link to /bin/dash. Dash is another shell (The 'D' is probably Debian).

The solution described is from linking /bin/sh to bash.

# 1) save copy of file '/bin/sh'  This is a symbolic link.
     [As a text file, it contains 4 charactic string 'dash']
#  -P, --no-dereference         never follow symbolic links in SOURCE
 sudo cp -P /bin/sh /bin/sh.orig
# remove /bin/sh
sudo rm /bin/sh
# make symbolic link from /bin/sh to /bin/bash
sudo ln -s /bin/bash /bin/sh

It is now the case that on this DO installation, sh xyz.sh executes the script using bash. In particular, the variable expansions such as ${var^^} are handled by bash shell. In the example case, the value in the variable named var is changed to upper case.

# put this into temp.sh
var='hello'
y="${var^^}"
echo "var=$var, y=$y"
# then execute with
sh temp.sh
### result:  var=hello, y=HELLO

funderburkjim commented 4 years ago

An alternate (Unused) solution.

In all our code scripts change sh xyz to bash xyz.

Probably the modified code would still work (a) on Cologne server and (b) under Git Bash in windows. The modified code would also likely work fine at DO.

The advantage of this method is that no change to Ubuntu installation is required. The disadvantage is that many of our code files would need to be changed.

funderburkjim commented 4 years ago

When can php write to file in DO setup?

When running via the web server, php program can NOT write to files. Thus, dbgprint.php won't be useful for debugging at DO. It is probably good security measure to retain this write limitation.
When running under ssh, php CAN write to files (within subdirectories of /var/www/html). See discussion of permissions above.

funderburkjim commented 4 years ago

Bash parameter expansion new solution

A third possibility seems better. The parameter expansion (to change variables to upper case or lower case) is really needed only in the generate_orig.sh script. In that script, we need the upper case dictionary code, and the caller of that script (generate_dict.sh) only has a lower case code. So, it is convenient for generate_orig.sh to use the bash parameter expansion ${dict^^ to get the upper case from lower case.

But we can execute this one script with 'bash' instead of with 'sh' under Ubuntu: bash generate_orig.sh .....
Conclusion: We can make our scripts compatible on original Ubuntu by

Execute generate_orig.sh script with bash (rather than sh)
Remove parameter expansion variables in generate_dict.sh, generate_pywork.sh and in csl-websanlexicon/v02/generate_web.sh
And continue to execute these 3 scripts (generate_dict.sh, etc.) with sh.

Result is

everything works properly under Ubuntu and
everything continues to work properly at Cologne and in GitBash terminal on Windows.

The original 'sh' on Ubuntu is now restored in the DO installation. This is better, since there is now no need for the additional Ubuntu installation step of the former solution.

gasyoun commented 4 years ago

@artforlife any clue?

YevgenJohn commented 4 years ago

I am trying to create a local Centos VM with the all the components to have a local version of what we see at www.sanskrit-lexicon.uni-koeln.de. I assume CSL* projects contain those. I wonder if there is any Wiki howto of the architecture of it. Does it use any sort of centralized DB, which one? Where we can get the DB dump? I would like to make a VM image one could use for a local use, with the ability to refresh it from the GitHub as the content is changing. It might potentially be reworked for Docker use, so something intensive could benefit clouds scaling. That would allow local use without a need to add to the webserver's load at Koeln. Digitized dictionaries open numerous parsing possibilities, which would impose a serious performance impact if run on a centralized web server opened to the public, so it is better to operate with a local VM image. Please advise where to look at to put things together. Thank you!

funderburkjim commented 4 years ago

@YevgenJohn

Take a look at https://github.com/sanskrit-lexicon/csl-pywork/tree/master/v02.

Let me know if documentation details are missing.

YevgenJohn commented 4 years ago

@YevgenJohn

Take a look at https://github.com/sanskrit-lexicon/csl-pywork/tree/master/v02.

Let me know if documentation details are missing.

Thank you, that worked. I installed XAMPP and software prerequisites and redo_xampp_all.sh made it functional. Great script! I checked search in MW, and it works just fine.

gasyoun commented 4 years ago

As @artforlife seems to be gone I recommend @YevgenJohn who has the server knowledge and is eager to help.

drdhaval2785 commented 4 years ago

@funderburkjim and @YevgenJohn Did a fresh installation on an ubuntu based distro. Have updated the installation instruction at https://github.com/sanskrit-lexicon/csl-pywork/tree/master/v02#initializationupdate-of-all-dictionaries-on-local-ubuntu-machine .

This may help someone interested in installing a local copy / ubuntu based server.

drdhaval2785 commented 4 years ago

@funderburkjim The installation works perfectly well on the local machine. So the idea of having everything on git seems to be on track. Best wishes.

YevgenJohn commented 4 years ago

@funderburkjim and @YevgenJohn Did a fresh installation on an ubuntu based distro. Have updated the installation instruction at https://github.com/sanskrit-lexicon/csl-pywork/tree/master/v02#initializationupdate-of-all-dictionaries-on-local-ubuntu-machine .

This may help someone interested in installing a local copy / ubuntu based server.

Thank you for the updated instruction. That's precisely what I've done except using XAMPP bundle (with a need of libnsl installed on Centos 7 to let lamp start) instead of apache2 only (wasn't sure if MySQL would be required somewhere). I guess the instruction is a good candidate for a Dockerbuild file at some future time. The design of it is so self-contained, so it can be done as Docker container, spawn on demand in a cloud on per-user case then discarded, so it can scale up perfectly without a centralized DB bottleneck.

funderburkjim commented 4 years ago

@YevgenJohn -

Just to confirm, mysql is not used. All databases are sqlite3.

YevgenJohn commented 4 years ago

Thank you, that's great design which makes them scale up better that without a single DB bottleneck, so each image of that software is self-contained. One might want mysql/Postrgres for some analytics with extensive parsing, but that is separate from the dictionary webpage functionality, and can easily be added to the scripts where it populates sqlite to also send it into some other DB, should one ever need a research version of it versus dictionary lookup.

YevgenJohn commented 4 years ago

Do we have a main page similarly to what is shown at https://www.sanskrit-lexicon.uni-koeln.de/ ? When we process all the dictionaries, we don't have the main page under apache, so we have to navigate to a separate dictionaries like http://localhost/cologne/mw/web/. We might need to add a step to modify links in there, in case it poinst to uni-koeln.de so local installation won't point to the Internet and is capable of operating offline. Please add it to the Git repo (or advise it's location, so we add this step to the instruction). Thank you!

funderburkjim commented 4 years ago

Re: home page for local installation

The home page construction is also in a repository:
https://github.com/sanskrit-lexicon/csl-homepage

Would this be a good place for a branch, to make versions 'make_index2_xampp.py' and 'redo_index_xampp.sh' appropriate for local (i.e., non-Cologne) homepage?

YevgenJohn commented 4 years ago

I'm getting familiar with different csl- repos, this is the 4th one (3 others were used to install dictionaries into the VM) in my attempt to understand the big architectural picture, thank you! The branch for local homepage is a really good use of it, one would be able to chose different branches following the same installation instruction and get customized solution depending on the branch selected. I like this approach, if that is convenient, thank you!

gasyoun commented 4 years ago

The home page construction is also in a repository: https://github.com/sanskrit-lexicon/csl-homepage

Hardly do I understand why it can't be part of another, bigger CSL. Not fit for Occam's razor.

YevgenJohn commented 4 years ago

Whichever way is more convenient to manage it, but there should be one master-list (index) which puts them all into a big picture altogether, explaining their interrelationship.

drdhaval2785 commented 2 years ago

The code works well for past two years. This documentation has lived its life. Closing this issue.

sanskrit-lexicon / csl-pywork