BS does not start on usaxscontrol

[jilavsky]

Setting up instrument, checking all starts...

Sounds like some databroker issue - was move to network location fished?

usaxs@usaxscontrol .../USAXS_data/2022-02 $ blueskyUSAXS 
Python 3.8.2 (default, Mar 26 2020, 15:53:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

IPython profile: bluesky
Exception reporting mode: Minimal
Activating auto-logging. Current session state plus future input saved.
Filename       : /share1/USAXS_data/2022-02/.logs/ipython_console.log
Mode           : rotate
Output logging : True
Raw input log  : False
Timestamping   : True
State          : active
I Fri-10:27:37 - ############################################################ startup
I Fri-10:27:37 - logging started
I Fri-10:27:37 - logging level = 10
I Fri-10:27:37 - /home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/collection.py
I Fri-10:27:37 - /home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/mpl/console.py
I Fri-10:27:37 - bluesky framework
I Fri-10:27:37 - /home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/framework/check_python.py
I Fri-10:27:37 - /home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/framework/initialize.py
2022-01-28 10:27:39,089 - intake - INFO - local.py:_load:L750 - Loading "name: databroker_catalogs
container: catalog
plugin: ['yaml_file_cat']
description: YAML file: databroker_catalogs
direct_access: True
user_parameters: []
metadata: 
args: 
  name: builtin
  ttl: 1
  path: /local/usaxs/.local/share/intake/databroker_catalogs.yml" as a catalog failed: [Errno 2] No such file or directory: '/local/usaxs/.local/share/intake/databroker_catalogs.yml'
2022-01-28 10:27:39,130 - intake - INFO - local.py:_load:L750 - Loading "name: databroker_catalogs
container: catalog
plugin: ['yaml_file_cat']
description: YAML file: databroker_catalogs
direct_access: True
user_parameters: []
metadata: 
args: 
  name: builtin
  ttl: 1
  path: /local/usaxs/.local/share/intake/databroker_catalogs.yml" as a catalog failed: [Errno 2] No such file or directory: '/local/usaxs/.local/share/intake/databroker_catalogs.yml'
2022-01-28 10:27:39,139 - intake - INFO - local.py:_load:L750 - Loading "name: databroker_catalogs
container: catalog
plugin: ['yaml_file_cat']
description: YAML file: databroker_catalogs
direct_access: True
user_parameters: []
metadata: 
args: 
  name: builtin
  ttl: 1
  path: /local/usaxs/.local/share/intake/databroker_catalogs.yml" as a catalog failed: [Errno 2] No such file or directory: '/local/usaxs/.local/share/intake/databroker_catalogs.yml'
2022-01-28 10:27:39,180 - intake - INFO - local.py:_load:L750 - Loading "name: databroker_catalogs
container: catalog
plugin: ['yaml_file_cat']
description: YAML file: databroker_catalogs
direct_access: True
user_parameters: []
metadata: 
args: 
  name: builtin
  ttl: 1
  path: /local/usaxs/.local/share/intake/databroker_catalogs.yml" as a catalog failed: [Errno 2] No such file or directory: '/local/usaxs/.local/share/intake/databroker_catalogs.yml'
2022-01-28 10:27:39,220 - intake - INFO - local.py:_load:L750 - Loading "name: databroker_catalogs
container: catalog
plugin: ['yaml_file_cat']
description: YAML file: databroker_catalogs
direct_access: True
user_parameters: []
metadata: 
args: 
  name: builtin
  ttl: 1
  path: /local/usaxs/.local/share/intake/databroker_catalogs.yml" as a catalog failed: [Errno 2] No such file or directory: '/local/usaxs/.local/share/intake/databroker_catalogs.yml'
[TerminalIPythonApp] WARNING | Unknown error in handling startup files:
KeyError: '9idc_usaxs'

[prjemian]

WHY? IS? USAXS? looking in the backup directory path /local/usaxs? Should be using ~ as root for this (~/.local/share/intake/ where the soft link in that directory works properly!).

[prjemian]

Fixed this with a workaround (remove the softlinks and copy the ~/.ipython/profile_bluesky/startup/databroker_catalogs.yml file into both the /local/usaxs/.local/share/intake and ~/.local/share/intake directories)

[prjemian]

It starts now with no additional errors. There is a known warning from the intake package; ignore that now since it's for the databroker developers to resolve.

[prjemian]

Note: The search path can be printed with these commands:

import databroker
databroker.catalog_search_path()

and the result (can't copy from the session now) shows the first item in the path is the /local/usaxs/.local/share/intake directory.

Q: How to change that to what was expected (~/.local/share/intake)?

[prjemian]

Interesting search. The intake package uses the appdirs.user_data_dir() function for this. Here's something important:

    Typical user data directories are:
        Mac OS X:               ~/Library/Application Support/<AppName>
        Unix:                   ~/.local/share/<AppName>    # or in $XDG_DATA_HOME, if defined
        Win XP (not roaming):   C:\Documents and Settings\<username>\Application Data\<AppAuthor>\<AppName>
        Win XP (roaming):       C:\Documents and Settings\<username>\Local Settings\Application Data\<AppAuthor>\<AppName>
        Win 7  (not roaming):   C:\Users\<username>\AppData\Local\<AppAuthor>\<AppName>
        Win 7  (roaming):       C:\Users\<username>\AppData\Roaming\<AppAuthor>\<AppName>```

[prjemian]

Checking the usaxs environment variables, there is a XDG_DATA_HOME environment variable and it is set to /local/share/.local/share. From this, appdirs appends /intake since that is the "application" being configured.

Note: None of this is the responsibility of bluesky. Wonder how and why this environment variable is set? @jilavsky - any ideas?

[prjemian]

Aha! It's set in /etc/profile.d/zzaps.sh. Pesky OS and sysAdmins!

[prjemian]

BTW: This is the same setup that creates the .loglogin files. That file was last modified in May 2019 but the edits look like they are from 2012. We can probably remove the env var in our setup.

[prjemian]

I confirm that will fix this issue:

unset XDG_DATA_HOME

[prjemian]

Just added that to the end of ~/.bashrc

This will be applied to new shells.

What is XDG_DATA_HOME and why is it relevant?

[jilavsky]

Reading this only increased level of my confusion. I am not responsible for any of those settings. May be Kevin?

[prjemian]

Nope. APS IT by the initials of JLM (namely Joe Midlock).

[prjemian]

XDG is where some system software stores local, user-dependent, configuration settings. Probably, Joe was reacting to problems where writing this info to $HOME/.local/share was complicated by NFS at the APS. His modifications redirect XDG to the /local/${USER} directory, knowing that APS IT always provides this local storage with the excess partition space on the hard disk.