search

xylucien / MusicalKits_Converter

My personal Flask project
1 stars 0 forks source link

music21 cannot load on server #62

Closed xylucien closed 4 years ago

xylucien commented 4 years ago 
[Wed Jul 22 00:29:22.084401 2020] [:error] [pid 6405] [remote 70.19.38.143:176] mod_wsgi (pid=6405): Target WSGI script '/opt/python/current/app/wsgi.py' cannot be loaded as Python module.
[Wed Jul 22 00:29:22.084476 2020] [:error] [pid 6405] [remote 70.19.38.143:176] mod_wsgi (pid=6405): Exception occurred processing WSGI script '/opt/python/current/app/wsgi.py'.
[Wed Jul 22 00:29:22.084645 2020] [:error] [pid 6405] [remote 70.19.38.143:176] Traceback (most recent call last):
[Wed Jul 22 00:29:22.084687 2020] [:error] [pid 6405] [remote 70.19.38.143:176]   File "/opt/python/current/app/wsgi.py", line 3, in <module>
[Wed Jul 22 00:29:22.084691 2020] [:error] [pid 6405] [remote 70.19.38.143:176]     application = create_app()
[Wed Jul 22 00:29:22.084697 2020] [:error] [pid 6405] [remote 70.19.38.143:176]   File "/opt/python/current/app/converter/__init__.py", line 57, in create_app
[Wed Jul 22 00:29:22.084701 2020] [:error] [pid 6405] [remote 70.19.38.143:176]     us['lilypondPath'] = stdout.decode().replace('\\n', '')
[Wed Jul 22 00:29:22.084706 2020] [:error] [pid 6405] [remote 70.19.38.143:176]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/music21/environment.py", line 1324, in __setitem__
[Wed Jul 22 00:29:22.084710 2020] [:error] [pid 6405] [remote 70.19.38.143:176]     self._environment.write()
[Wed Jul 22 00:29:22.084715 2020] [:error] [pid 6405] [remote 70.19.38.143:176]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/music21/environment.py", line 1134, in write
[Wed Jul 22 00:29:22.084718 2020] [:error] [pid 6405] [remote 70.19.38.143:176]     return envSingleton().write(filePath=filePath)
[Wed Jul 22 00:29:22.084724 2020] [:error] [pid 6405] [remote 70.19.38.143:176]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/music21/environment.py", line 781, in write
[Wed Jul 22 00:29:22.084727 2020] [:error] [pid 6405] [remote 70.19.38.143:176]     raise EnvironmentException('bad file path for .music21rc: %s' % filePath)
[Wed Jul 22 00:29:22.084743 2020] [:error] [pid 6405] [remote 70.19.38.143:176] music21.environment.EnvironmentException: bad file path for .music21rc: /home/wsgi/.music21rc
xylucien commented 4 years ago

it seems that eb server has different path structure, and the script cannot find .music21rc, where the music21 configuration is stored

xylucien commented 4 years ago 
[Wed Jul 22 00:48:49.558776 2020] [:error] [pid 7248] [remote 70.19.38.143:148] Traceback (most recent call last):
[Wed Jul 22 00:48:49.558818 2020] [:error] [pid 7248] [remote 70.19.38.143:148]   File "/opt/python/current/app/wsgi.py", line 3, in <module>
[Wed Jul 22 00:48:49.558822 2020] [:error] [pid 7248] [remote 70.19.38.143:148]     application = create_app()
[Wed Jul 22 00:48:49.558828 2020] [:error] [pid 7248] [remote 70.19.38.143:148]   File "/opt/python/current/app/converter/__init__.py", line 56, in create_app
[Wed Jul 22 00:48:49.558832 2020] [:error] [pid 7248] [remote 70.19.38.143:148]     environment.set('lilypondPath', stdout.decode().replace('\\n', ''))
[Wed Jul 22 00:48:49.558838 2020] [:error] [pid 7248] [remote 70.19.38.143:148]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/music21/environment.py", line 1399, in set
[Wed Jul 22 00:48:49.558842 2020] [:error] [pid 7248] [remote 70.19.38.143:148]     us.create()  # no problem if this does not exist
[Wed Jul 22 00:48:49.558847 2020] [:error] [pid 7248] [remote 70.19.38.143:148]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/music21/environment.py", line 1335, in create
[Wed Jul 22 00:48:49.558860 2020] [:error] [pid 7248] [remote 70.19.38.143:148]     self._environment.write()
[Wed Jul 22 00:48:49.558866 2020] [:error] [pid 7248] [remote 70.19.38.143:148]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/music21/environment.py", line 1134, in write
[Wed Jul 22 00:48:49.558870 2020] [:error] [pid 7248] [remote 70.19.38.143:148]     return envSingleton().write(filePath=filePath)
[Wed Jul 22 00:48:49.558875 2020] [:error] [pid 7248] [remote 70.19.38.143:148]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/music21/environment.py", line 781, in write
[Wed Jul 22 00:48:49.558893 2020] [:error] [pid 7248] [remote 70.19.38.143:148]     raise EnvironmentException('bad file path for .music21rc: %s' % filePath)
[Wed Jul 22 00:48:49.558912 2020] [:error] [pid 7248] [remote 70.19.38.143:148] music21.environment.EnvironmentException: bad file path for .music21rc: /home/wsgi/.music21rc
xylucien commented 4 years ago

the problem is to locate .music21sc on eb server. According to music21 docs:

Environment configuration files are stored in platform-specific locations. On Linux and MacOS computers, the configuration file is stored as the file .music21rc, located in the user’s home directory.

xicheng87 commented 4 years ago

Does this mean somehow .music21rc was not created? Maybe look into music21/environment.py and see where this EnvironmetException is thrown?

xylucien commented 4 years ago

The path to the environment settings file can always be found with the getSettingsPath() method.

us = environment.UserSettings() us.getSettingsPath() '/Users/cuthbert/.music21rc'

xicheng87 commented 4 years ago

But why is the exception thrown in by music21?

xylucien commented 4 years ago

During the app init, I ask the server to write lilypond location into .music21rc in order to make visualization working.

xylucien commented 4 years ago

I implement this by calling environment.set, which is a function from music21

xylucien commented 4 years ago

An idea: music21 asks users to run configure.run() before using, which is what I missed.

xylucien commented 4 years ago

configure.run() requires user input during its processe, I dont know what to do with that. Should I just create the file manually by os.write() instead?

xicheng87 commented 4 years ago

Yeah, it sounds like a hack to manually create the file but I dont' have other great ideas. You have two options: 1) manually create on with the Procfile or 2) create it with python script.

xylucien commented 4 years ago

I tried to add this line to procfile but it didnt work: command: echo '<settings encoding="utf-8"><preference name="autoDownload" value="deny" /><preference name="braillePath" /><preference name="debug" value="0" /><preference name="directoryScratch" /><preference name="graphicsPath" value="/usr/bin/eog" /><preference name="ipythonShowFormat" value="ipython.musicxml.png" /><preference name="lilypondBackend" value="ps" /><preference name="lilypondFormat" value="pdf" /><preference name="lilypondPath" value="/usr/bin/lilypond" /><preference name="lilypondVersion" /><localCorporaSettings /><localCorpusSettings /><preference name="manualCoreCorpusPath" /><preference name="midiPath" /><preference name="musescoreDirectPNGPath" /><preference name="musicxmlPath" value="/usr/bin/musescore" /><preference name="pdfPath" /><preference name="showFormat" value="musicxml" /><preference name="vectorPath" /><preference name="warnings" value="1" /><preference name="writeFormat" value="musicxml" /></settings>' > /home/wsgi/.music21rc

xylucien commented 4 years ago

python approach failed as well. I am guessing that /home/wsgi/.music21rc seems to be an invalid path

xicheng87 commented 4 years ago

Ok, let's revert these changes and rethink. So it seems that on AWS EB, the username is "wsgi" so it expects a .music21rc there right?

xylucien commented 4 years ago

right

xicheng87 commented 4 years ago

I took a look at the environment.py and the relevant code snippet:

    def write(self, filePath=None):
        '''
        Write the .music21rc using `.getSettingsPath` and `.toSettingsXML`.
        '''
        if filePath is None:
            filePath = self.getSettingsPath()
        # need to use __getitem__ here b/c need to convert debug value
        # to an integer
        if filePath is None or not filePath.parent.exists():
            raise EnvironmentException('bad file path for .music21rc: %s' % filePath)
        settingsTree = self.toSettingsXML()
        etIndent(settingsTree.getroot())

So from the error message, we know that filePath = /home/wsgi/.music21rc, and the exception condition above seems to indicate not filePath.parent.exists() = true. Do you agree? I guess this has to do with AWS EB environment?

xylucien commented 4 years ago

yeh, I agree with that. so should I also create the folder /home/wsgi/?

xicheng87 commented 4 years ago

Better understand why self.getSettingsPath() returns /home/wsgi/.music21rc at the first place? The call stack indicates that filePath is not set initially when this is called, so self.getSettingsPath() must have called.

xylucien commented 4 years ago

in self.getSettingsPath():

 elif platform in ['nix', 'darwin']:
            # might not exist if running as nobody in a webserver...
            if 'HOME' in os.environ:
                directory = pathlib.Path(os.environ['HOME'])
            else:
                directory = pathlib.Path('/tmp/')
            return directory / '.music21rc'

so if the console returns /home/wsgi/.music21rc, it means that /home/wsgi exists on eb server. Then I don't understand why my method has failed.

xicheng87 commented 4 years ago

I think this means that os.environ['HOME'] points to a path named /home/wsgi, but is there any code that checks whether this path exists? It is certainly absurd if HOME is set but points to a non-existing path, but we may be surprised.

Is there other debugging information we can obtain from the AWS EB?

xylucien commented 4 years ago

I put on a new part which asks the script to create os.environ['HOME'] folder if doesnt exist. lets see what will happen.

xylucien commented 4 years ago 
[Thu Jul 23 01:34:19.228377 2020] [:error] [pid 23191] [remote 70.19.38.143:132] mod_wsgi (pid=23191): Target WSGI script '/opt/python/current/app/wsgi.py' cannot be loaded as Python module.
[Thu Jul 23 01:34:19.228441 2020] [:error] [pid 23191] [remote 70.19.38.143:132] mod_wsgi (pid=23191): Exception occurred processing WSGI script '/opt/python/current/app/wsgi.py'.
[Thu Jul 23 01:34:19.228551 2020] [:error] [pid 23191] [remote 70.19.38.143:132] Traceback (most recent call last):
[Thu Jul 23 01:34:19.228582 2020] [:error] [pid 23191] [remote 70.19.38.143:132]   File "/opt/python/current/app/wsgi.py", line 3, in <module>
[Thu Jul 23 01:34:19.228587 2020] [:error] [pid 23191] [remote 70.19.38.143:132]     application = create_app()
[Thu Jul 23 01:34:19.228593 2020] [:error] [pid 23191] [remote 70.19.38.143:132]   File "/opt/python/current/app/converter/__init__.py", line 47, in create_app
[Thu Jul 23 01:34:19.228596 2020] [:error] [pid 23191] [remote 70.19.38.143:132]     setupAppAndCacheDirectories(app)
[Thu Jul 23 01:34:19.228602 2020] [:error] [pid 23191] [remote 70.19.38.143:132]   File "/opt/python/current/app/converter/__init__.py", line 22, in setupAppAndCacheDirectories
[Thu Jul 23 01:34:19.228605 2020] [:error] [pid 23191] [remote 70.19.38.143:132]     os.makedirs(os.environ['HOME'])
[Thu Jul 23 01:34:19.228610 2020] [:error] [pid 23191] [remote 70.19.38.143:132]   File "/opt/python/run/venv/lib64/python3.6/os.py", line 220, in makedirs
[Thu Jul 23 01:34:19.228614 2020] [:error] [pid 23191] [remote 70.19.38.143:132]     mkdir(name, mode)
[Thu Jul 23 01:34:19.228628 2020] [:error] [pid 23191] [remote 70.19.38.143:132] PermissionError: [Errno 13] Permission denied: '/home/wsgi'
xylucien commented 4 years ago

cant really do that from the scirpt, will try profile instead.

xicheng87 commented 4 years ago

I don't think you can create a folder under /home for permission reasons, because the subdir under it corresponds to different users. One thing you may try is to simply crash the script if the folder doesn't exist, that way you'd know what is going on in the deployment env.

xicheng87 commented 4 years ago

Looks that the latest change doesn't work either. Can you revert some of the latest ones, and add the checking of os.environ['HOME'] folder and crash if it doesn't exist? Let's rethink what is going on.

xylucien commented 4 years ago

will try that.

xylucien commented 4 years ago 
[Thu Jul 23 02:28:31.353003 2020] [:error] [pid 25329] [remote 70.19.38.143:55684] mod_wsgi (pid=25329): Target WSGI script '/opt/python/current/app/wsgi.py' cannot be loaded as Python module.
[Thu Jul 23 02:28:31.353052 2020] [:error] [pid 25329] [remote 70.19.38.143:55684] mod_wsgi (pid=25329): Exception occurred processing WSGI script '/opt/python/current/app/wsgi.py'.
[Thu Jul 23 02:28:31.353141 2020] [:error] [pid 25329] [remote 70.19.38.143:55684] Traceback (most recent call last):
[Thu Jul 23 02:28:31.353170 2020] [:error] [pid 25329] [remote 70.19.38.143:55684]   File "/opt/python/current/app/wsgi.py", line 3, in <module>
[Thu Jul 23 02:28:31.353175 2020] [:error] [pid 25329] [remote 70.19.38.143:55684]     application = create_app()
[Thu Jul 23 02:28:31.353181 2020] [:error] [pid 25329] [remote 70.19.38.143:55684]   File "/opt/python/current/app/converter/__init__.py", line 44, in create_app
[Thu Jul 23 02:28:31.353185 2020] [:error] [pid 25329] [remote 70.19.38.143:55684]     setupAppAndCacheDirectories(app)
[Thu Jul 23 02:28:31.353191 2020] [:error] [pid 25329] [remote 70.19.38.143:55684]   File "/opt/python/current/app/converter/__init__.py", line 22, in setupAppAndCacheDirectories
[Thu Jul 23 02:28:31.353195 2020] [:error] [pid 25329] [remote 70.19.38.143:55684]     raise
[Thu Jul 23 02:28:31.353208 2020] [:error] [pid 25329] [remote 70.19.38.143:55684] RuntimeError: No active exception to reraise
xylucien commented 4 years ago

so it does not exist.

xicheng87 commented 4 years ago

OK, this confirms our hypothesis. Next question, how could os.environ['HOME'] becomes "wsgi"? Has this to do with the wsgi configuration in flask?

xicheng87 commented 4 years ago

And what is the os.environ['HOME'] value on your local run? This should really correspond to the username of the one that runs the program on the machine.

xylucien commented 4 years ago

os.environ['HOME'] '/home/lucien'

xylucien commented 4 years ago

lucien is the username i set when i was installing ubuntu, which is also the one I used to run the program.

xylucien commented 4 years ago

I managed to ssh to the server. Interestingly enough, when I run os.environ['HOME'] it returns me 

>>> os.environ['HOME']
'/home/ec2-user'

because my username is ec2-user

xylucien commented 4 years ago

It appears to be clear now that everything is being executed under /opt/python/current/app with username wsgi, but the folder /home/wsgi is never created, which causes the bug.

Obviously, the creator of music21 did not ever think of putting this on an AWS EB server...

xicheng87 commented 4 years ago

So, this means /opt/python/current/app overrides the os.environ['HOME'] to be wsgi right? Where did this happen, and can we revise this env var ourself?

xylucien commented 4 years ago

basically yes. os.environ['HOME'] depends on the current user. For example it is /home/lucien if I run it locally. But if I change to super user mode (locally as well) it would be /root.

xylucien commented 4 years ago

maybe try to just set the os.environ['HOME'] to the current working directory (cuz its guaranteed to exist?)

update: it works locally.

xicheng87 commented 4 years ago

Yes, let's give it a try - I was just curious how it got changed to a non-existing user by app

xylucien commented 4 years ago

works. Another bugs happened tho