Branch personal/Dennis walkthrough

[Steven-Harris]

Finally, there is a detailed tutorial on how to use rapuma! It is very easy to understand and use. I did want to start out by showing you an error I got when starting the tutorial. It appears rapuma is having the error when I add a group of type usfm. When I ran the command rapuma content KYU-MYMR-KYUMTEST group add --group GOSPEL --comp_type usfm I got

Traceback (most recent call last): File "/usr/local/bin/rapuma", line 895, in userArguments(parser.parse_args()) File "/usr/local/bin/rapuma", line 456, in userArguments ProjSetup(sysConfig, pid, gid).addGroup(cType, gid) File "/usr/local/lib/python2.7/dist-packages/rapuma/project/proj_setup.py", line 306, in addGroup aProject.createGroup() File "/usr/local/lib/python2.7/dist-packages/rapuma/manager/project.py", line 202, in createGroup groupObj = ManagerClass(self, cfg) File "/usr/local/lib/python2.7/dist-packages/rapuma/group/usfm.py", line 60, in init self.proj_illustration = ProjIllustration(self.pid, self.gid) File "/usr/local/lib/python2.7/dist-packages/rapuma/project/proj_illustration.py", line 60, in init self.userIllustrationLibName = self.userConfig['Resources']['defaultIllustrationLibraryName'] File "/usr/lib/python2.7/dist-packages/configobj.py", line 554, in getitem val = dict.getitem(self, key) KeyError: 'defaultIllustrationLibraryName'

Although, it appears to be working fine still after

[Steven-Harris]

Another problem when I was going through the tutorial was when I run rapuma content KYU-MYMR-KYUMTEST component add --group GOSPEL --cid_list "mat mrk luk jhn act" --path ~/Publishing/my_source/KYUM/PT-source I got

MSG - Importing: [Matthew] Traceback (most recent call last): File "/usr/local/bin/rapuma", line 895, in userArguments(parser.parse_args()) File "/usr/local/bin/rapuma", line 469, in userArguments ProjSetup(sysConfig, pid, gid).addComponent(gid, source_list) File "/usr/local/lib/python2.7/dist-packages/rapuma/project/proj_setup.py", line 378, in addComponent return handleUsfmAdd() File "/usr/local/lib/python2.7/dist-packages/rapuma/project/proj_setup.py", line 333, in handleUsfmAdd if self.installGroupComps(gid, sourceList) : File "/usr/local/lib/python2.7/dist-packages/rapuma/project/proj_setup.py", line 524, in installGroupComps if self.importUsfmWorkingText(fName, cType, gid, cid) : File "/usr/local/lib/python2.7/dist-packages/rapuma/project/proj_setup.py", line 711, in importUsfmWorkingText if self.usfmCopy(compFiles['source'], compFiles['working'], gid) : File "/usr/local/lib/python2.7/dist-packages/rapuma/project/proj_setup.py", line 785, in usfmCopy sourceEncode = self.projectConfig['Managers']['usfm_Text']['sourceEncode'] File "/usr/lib/python2.7/dist-packages/configobj.py", line 554, in getitem val = dict.getitem(self, key) KeyError: 'usfm_Text'

I believe this issues are probably related since the GOSPEL group was not successfully added

[thresherdj]

I'll be on the road for a while so I'll try to work through this tonight and get you some answers. Thanks!

[thresherdj]

So, on the first error: KeyError: 'defaultIllustrationLibraryName' I need to ask you to take a look at the ~/config/rapuma/rapuma.conf file around line 16, it should have this: defaultIllustrationLibraryName = Knowles-600_bw If it isn't there that is causing the first error. Something in the Rapuma setup didn't work right.

On the second error, look in the project.conf file around line 56 there should be (but probably isn't) a sub section called [usfm_Text]. If that isn't there, that might be due to the first error. You could go back and rerun: rapuma content KYU-MYMR-KYUMTEST group add --group GOSPEL --comp_type usfm After that you should be able to add the components. Let me know how it goes.

[Steven-Harris]

Yep, I do not have the defaultillustratoinlibraryname in the ~/.config I tried adding that line in manually but I did get around the error from the first time

[Steven-Harris]

Hmm that's interesting.... Originally I had rapuma running in a VM for these test. However running rapuma on my local machine I had to problems or errors creating the groups.

[thresherdj]

Not sure why that would be. We've run Rapuma in production on a VM and didn't seem to see any difference.

[Steven-Harris]

Good news! This version of rapuma is awesome and I was able to make it through your example with no problems. I did have one question that came up when I installed rapuma for the first time and that is.... what does this command translate to in the new app.. rapuma system setup projects -s "~/Publishing"

[thresherdj]

That's great, I'm glad you like it. I don't have time just now to explain what is different in the command you cited. You might be able to figure it out from the --help feature, but maybe not. It may not be too intuitive. It involves giving Rapuma the section and value you want to change along with the new value. It is more generalized than the earlier version.

[Steven-Harris]

Ok, I could figure out what section the Rapuma settings fell under since they were not part of a project of normal project commands On Wed, Jan 27, 2016 at 7:16 AM Dennis Drescher notifications@github.com wrote:

That's great, I'm glad you like it. I don't have time just now to explain what is different in the command you cited. You might be able to figure it out from the --help feature, but maybe not. It may not be too intuitive. It involves giving Rapuma the section and value you want to change along with the new value. It is more generalized than the earlier version.

— Reply to this email directly or view it on GitHub https://github.com/sillsdev/rapuma/issues/3#issuecomment-175618560.

[Steven-Harris]

Couldn't^

[thresherdj]

The confusing part is that you need to have a project ID (valid) to make system-wide settings. That being the case, the command you would need would be this:

rapuma setting rapuma --section Resources --key Projects --value "~/Publishing"

However, I can see a problem here. If you are going to change some system settings before you actually do a project, you are stuck, unless you manually edit the conf file. That will not be very good. I'm thinking about changing this so that if you put in a key (special) word, it will change the system conf file without there being any project present. I'm thinking I would use "SYSTEM" in the project ID spot. What do you think?

[Steven-Harris]

We mainly just need to tell Rapuma where the Publishing folder and resources are. So that both apps can work together smoothly and our web app doesn't have to a make a call to around directory other than its own On Wed, Jan 27, 2016 at 5:39 PM Dennis Drescher notifications@github.com wrote:

The confusing part is that you need to have a project ID (valid) to make system-wide settings. That being the case, the command you would need would be this:

rapuma setting rapuma --section Resources --key Projects --value "~/Publishing"

However, I can see a problem here. If you are going to change some system settings before you actually do a project, you are stuck, unless you manually edit the conf file. That will not be very good. I'm thinking about changing this so that if you put in a key (special) word, it will change the system conf file without there being any project present. I'm thinking I would use "SYSTEM" in the project ID spot. What do you think?

— Reply to this email directly or view it on GitHub https://github.com/sillsdev/rapuma/issues/3#issuecomment-175911949.

[thresherdj]

Ok, I'll get that fixed as soon as I can. It shouldn't be hard, just need a few minutes to do it.

[thresherdj]

I got it fixed. You can now use the key word 'system' to make a change to a Rapuma system setting file. I pushed it to the personal/dennis branch and updated the version number. If you can confirm that you are going to be using this version for your project, I can merge it with the master. (At least I think I can merge it. I'm not real good at that kind of thing.) Then I can fix any bugs that pop up.

[Steven-Harris]

Yep! we are going to use your version.

[Steven-Harris]

Does this command work yet? rapuma setting rapuma --section Resources --key Projects --value "~/Publishing"

[thresherdj]

No, that will not work because you are missing the value in the project ID spot. I have fixed this so that for system-wide config changes you use the "SYSTEM" key word/value. So, the syntax would go like this:

rapuma setting SYSTEM rapuma --section Resources --key projects --value "~/Publishing"

BTW, the SYSTEM key word is upper case for emphasis, Any case should work, "system", "System", "SyStEm", ect.

It should work after you update. Let me know if it doesn't. BTW, I've updated the documentation as well but finding this tid-bit of information might be straightforward.

[thresherdj]

might ^not be straightforward

[Steven-Harris]

I having a pretty weird issue that is a little hard to explain because I am not entirely sure what is going on but I will do my best to explain. I am hoping you will have some insight as to what I could try next..

I am trying to get a php wrapper setup for rapuma using the built in "shell_exec" function. Of course when I run this command in the server environment it is running as the "www-data" user. That is causing quite a bit of problems with permissions... I have made www-data the owner of the /var/www folder and I pointing to /usr/local/bin/rapuma for the execution. My php script is getting caught up on line 119 of the rapuma script that locates the Userconfig(). I am a little lost on how to point it to the right config. I noticed my script created a folder called ".config" under /var/www/ - which is probably why it wanted access... In the ".config" there is now a rapuma.config file with the following contents.

[System] userName = Default User debugging = False svgPdfConvertCommand = rsvg-convert, -f, pdf, -o, pdfFile, svgFile projLogLineLimit = 1000 initDate = 2016-02-07 16:14:24 userID = default_user langID = ENG pdfViewerCommand = evince, -w pdfUtilityCommand = pdftk, textDifferentialViewerCommand = meld, autoHelperScripts = False recognizedComponentTypes = usfm, map maxStoreBackups = 5 projects = /home/supermam/src/web-scriptureforge/Publishing

of course I change the default value in project to point to my publishing folder.. but the rapuma script is still erroring out on line 119 getting the userconfig.

Any insight that you can give to what the problem might be would be greatly appreciated

[megahirt]

Steve,

I had a problem like this on a different project. Rapuma expects to run as a real user. www-data is not a real user. IMO you don't want a .config folder under /var/www . This is just a symptom of the problem. One way to get around this is to change the user that apache runs as. This will take a bit of tweaking but I've done it on my machine for a different project. I assume you have a user 'steve' on your machine.

1) To start with, add 'steve' to the www-data group. 2) change the user and group of apache in /etc/apache2/envvars 3) restart apache 'sudo service apache2 restart'

If you have issues with php and caching, I also had to chown a php cache directory on my machine to be root, but YMMV

[thresherdj]

Hi Steve,

I confess that I'm not too strong in understanding how rapuma interacts with the system when being run as a web-based app. What Chris told you sounds like an important set of steps. I also should mention that you will need a user config file here: /var/librapuma/config/rapuma.conf. I'm pretty sure that if you don't have a config file there it is going to choke and the error at line 119 would seem to indicate that, but I might be wrong. As I recall, you have to manually install the config file there. Let us know how it goes.

[Steven-Harris]

@megahirt, I have been working on this for a while and unfortunately I haven't got a working solution yet. Here is what I have done so far... 1) add my user to www-data with sudo usermod -G www-data superman 2) changed the user of apache from www-data to superman with sudo nano /etc/apache2/envvars and changing the appropriate line.

Doing this really messed up scriptureforge so I had to change it back.

3) it goes with out saying I restarted apache after doing this changes. 4) I messed around with @thresherdj suggestion and added a rapuma.config to /var/lib/rapuma/config and /usr/lib/rapuma/config/ (just for fun). I was using the rapuma.config that was generated from when I had a .config folder under /var/www/.config/.. I could not find one to use otherwise and I didn't know if I could just simply sudo touch /var/lib/rapuma/config/rapuma.config

Let me know if you have any other suggestions.

Thanks,

[thresherdj]

Hey Steven, most of this is above my pay-grade but your idea to create a file in /var/lib/rapuma/config/ may get a file there but for Rapuma to work it will need to be populated with the right data, but you probably knew that. I would think that the one from /var/www/.config/ would work just as well.

[megahirt]

Steve,

I see two ways forward. As of right now I believe Rapuma expects to be run as a real user with a writable home directory. This means that there will always be problems running Rapuma as www-data until such time that Rapuma no longer expects to have a writable home directory. I believe Dennis already removed some of this requirement by having a system-level config, presumably defined in /var/lib/rapuma/config/ Dennis will have to weigh in on how much of the current Rapuma code base is dependent on a user home directory.

The other way forward is to make apache run as a real user. I succeeded in this for my project, and I think you can too. You are going to need to look into the exact error message that scriptureforge is producing to appear "messed up". In my case, there was an error with PHP not being able to write to its own cache file, and I fixed it by changing the owner and permissions on that directory. After that, scriptureforge ran just fine for me locally.

[thresherdj]

With the introduction of the web interface and projects needing to be managed in the www environment, I added the possibility of having two places where the rapuma.conf file could be. If you need it to be looking in /var/lib/rapuma/config for that file, I could do that. One way forward would be to have that file "hardened" on install so that it would no longer need to be writeable. Then Rapuma could work anywhere on the system and there would be no security issues with read/writes in /var/lib/. Does that make sense? Should I look into doing that?

[Steven-Harris]

Yes! I think you should On Mon, Feb 15, 2016 at 4:59 AM Dennis Drescher notifications@github.com wrote:

With the introduction of the web interface and projects needing to be managed in the www environment, I added the possibility of having two places where the rapuma.conf file could be. If you need it to be looking in /var/lib/rapuma/config for that file, I could do that. One way forward would be to have that file "hardened" on install so that it would no longer need to be writeable. Then Rapuma could work anywhere on the system and there would be no security issues with read/writes in /var/lib/. Does that make sense? Should I look into doing that?

— Reply to this email directly or view it on GitHub https://github.com/sillsdev/rapuma/issues/3#issuecomment-184166531.

[Steven-Harris]

@megahirt, update. I change the apache user to run as me now with no problems with scriptureforge. However, rapuma is still erroring out on line 119 of /usr/local/bin/rapuma.

[Steven-Harris]

@megahirt and @thresherdj Update! I was able to get rapuma to execute rapuma -h successfully after changing the apache runtime user to myself! I noticed that some of my users setting were changed around to much when I was experimenting with user and group premissions.

[thresherdj]

Glad you got around/through this. I still will try to rework the way Rapuma sees its config file. Not sure how long that will take but I'll try.