Error when starting BLACS from console

[philipstarkey]

Original report (archived issue) by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

When I cd to the labscript_suite folder and then type in python blacs I get the following error:

#!python

Traceback (most recent call last):
  File "/Users/janwerkmann/anaconda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/janwerkmann/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/janwerkmann/labscript_suite/blacs/__main__.py", line 689, in <module>
    app = BLACS(qapplication)
  File "/Users/janwerkmann/labscript_suite/blacs/__main__.py", line 289, in __init__
    self.analysis_submission = AnalysisSubmission(self,self.ui)
  File "blacs/analysis_submission.py", line 37, in __init__
    self._ui = UiLoader().load(os.path.join(os.path.dirname(os.path.realpath(__file__)),'analysis_submission.ui'))
  File "/Users/janwerkmann/anaconda/lib/python2.7/site-packages/qtutils/UiLoader.py", line 118, in load
    return uic.loadUi(*args, **kwargs)
  File "/Users/janwerkmann/anaconda/lib/python2.7/site-packages/PyQt4/uic/__init__.py", line 236, in loadUi
    return DynamicUILoader(package).loadUi(uifile, baseinstance, resource_suffix)
  File "/Users/janwerkmann/anaconda/lib/python2.7/site-packages/PyQt4/uic/Loader/loader.py", line 71, in loadUi
    return self.parse(filename, resource_suffix, basedir)
  File "/Users/janwerkmann/anaconda/lib/python2.7/site-packages/PyQt4/uic/uiparser.py", line 974, in parse
    document = parse(filename)
  File "/Users/janwerkmann/anaconda/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse
    tree.parse(source, parser)
  File "/Users/janwerkmann/anaconda/lib/python2.7/xml/etree/ElementTree.py", line 647, in parse
    source = open(source, "rb")
IOError: [Errno 2] No such file or directory: '/Users/janwerkmann/labscript_suite/blacs/blacs/analysis_submission.ui'

what fixes this for me is in analysis_submission.ui replacing

#!python
        self._ui = UiLoader().load(os.path.join(os.path.dirname(os.path.realpath(__file__)),'analysis_submission.ui'))

with:

#!python

self._ui = UiLoader().load('analysis_submission.ui')

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: pstarkey).

---

Well the correct way to launch BLACS should really be python -m blacs, however that results in a different error for me (relating to importing device classes from labscript_devices)

Looking into this a bit more, I feel like something is kind of wrong with python. What version of Python are you running? I can replicate the behaviour with 2.7.13. It seems to me that os.path.realpath doesn't work properly. It seems to just append the contents of __file__ to the directory that __file__ is in, but __file__ also contains some of the directories as well if the current working directory is not the location of __file__. The same thing seems to happen with abspath too.....

So it could be fixed by combining the value of file with the cwd but that seems very hacky. I feel like I've misunderstood how Python paths work, or something has changed in Python. There are a lot of places where we use code exactly like this, so we'll have to fix all of them I think which is annoying!

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

This only happens when you run it from that directory - if you run it from elsewhere with 'python -m blacs' then this problem doesn't occur. Nonetheless I didn't realise that was a problem, and I should fix it. It indicates I was doing something wrong anyway...and this explains why sometimes log files appear in lyse/lyse/lyse.log ...

BLACS does do a os.chdir(os.path.abspath(os.path.dirname(__file__))) at the top of its __main__ file, so 'analysis_submission.ui' should be fine.

By a surprising coincidence, when I tested running BLACS with python -m blacs from a different directory, I got an error with zprocess trying to start child processes:

/usr/bin/python: can't open file '/home/bilbo/labscript_suite/blacs/zprocess/process_class_wrapper.py': [Errno 2] No such file or directory

This appears to be the exact same issue - I was in '/usr/local/lib/python2.7/dist-packages' having just purged qtutils from there, and zprocess lives in that directory too.

I've seen error messages before where I've installed a package from source, immediately opened a python interpreter in that directory and tested if I could import it, and had everything horribly fail. One project was kind enough to check that I was still in the source directory and raised an error saying "this won't work in this directory - please go somewhere else"

And if I print(__file__) in analysis_submission.py when launched from the labscript install directory, I get '/home/bilbo/labscript_suite/blacs/blacs/analysis_submission.pyc', which is less than helpful.

And with your fix, if I run "python -m blacs" from the labscript suite directory, I get import errors about labscript devices.

I'll look into it, but the problem stems from all the __file__ attributes being relative paths (which are not affected by os.chdir()) and are therefore wrong after an os.chdir()) if you are in the labscript suite directory, whereas if you are elsewhere they are absolute paths. So we have to either not use __file__ and rely on being in the right directory already, or we have to use __file__, but not do an os.chdir(). Not doing os.chdir() ever is not really an option I think - lots of things expect to be able to do it, if only temporarily.

So I'll look into it but I think what we probably want is to have something that always gets imported, like labscript_utils, check if the __main__ module's __file__ attribute is a relative path rather than an absoluite one, and throw an error saying "sorry, don't do that from the labscript suite directory".

[philipstarkey]

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

I'm on 2.7.13.

JWrkPC:~ janwerkmann$ python -V
Python 2.7.13 :: Continuum Analytics, Inc.

[philipstarkey]

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

Thanks for looking into it. Didn't think it was much of a problem until now nor did I know I was starting all my labscript apps 'wrong'. Btw what does work is doing python __main__.py and python FULLPATH/blacs/__main__.py where as python blacs/__main__.py doesn't work. Why exactly shouldn't I run things directly from the directory?

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Really? python __main__.py, if I start in the blacs directory, does work for me - though python blacs/__main__.py from the folder above does not.

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: pstarkey).

---

@cbillington FYI, if it wasn't obvious from my above post, launching like that from any higher up directory doesn't work either (you get the relative path of __file__ appended to absolute path of the BLACS folder)

[philipstarkey]

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

My bad I accidentally got the two commands mixed up.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

@pstarkey if I understand correctly, launching without the '-m' flag will fail from any higher directory. Launching with the '-m' flag will fail only if you are in a directory that contains some of the Python modules you are importing. Launching with the '-m' flag from one or more directories higher is fine. For example I usually run "python -m blacs" from my home directory, even though labscript_suite is in my home directory as well.

This is because Python adds the current directory to the module search path, so if it finds any modules in the current directory, their __file__ attributes will be relative paths, which become wrong as soon as a chdir() is run.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Python eventually considered this a bug and it was fixed in python 3.4. __file__ is always absolute in python ≥ 3.4 with the sole exception of a main module executed with a relative path.

I suspect any workaround for Python 2.7 is probably more trouble than it's worth, so until 2020 we can just say that python -m blacs from a directory not containing Python modules or python __main__.py in the BLACS directory are the only supported ways to start BLACS (same goes for the other programs). The launchers on Windows do the latter. We should make launchers for the other platforms too - it's not hard on linux, I don't know why I haven't already done it already.

[philipstarkey]

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

You are probably right. I'll try making a launcher for macOS once I figure out how that works.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

• changed state from "new" to "wontfix"