update instrument configuration to current practice

[prjemian]

Time to restart bluesky operations at 29-ID. There is a new recommendation to layout the .ipython directory (see 3-ID and 8-ID-I XPCS). this should be easier to understand and maintain.

[prjemian]

No need to work in a branch since there is NO configuration now. The beam line account already has a ~/.ipython directory. Install new configuration into ~/.ipython-bluesky directory.

The starter script (for ~/bin/) will be helpful to pull this all together.

[prjemian]

Looking good:

(bluesky_2020_1) [sleepy /tmp] mkdir ~/.ipython-bluesky
(bluesky_2020_1) [sleepy /tmp] cd ~/.ipython-bluesky/
(bluesky_2020_1) [sleepy ~/.ipython-bluesky] git clone https://github.com/APS-29ID-IEX/ipython-29id ./
Cloning into '.'...
remote: Enumerating objects: 43, done.
remote: Counting objects: 100% (43/43), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 167 (delta 5), reused 39 (delta 4), pack-reused 124
Receiving objects: 100% (167/167), 57.86 KiB | 0 bytes/s, done.
Resolving deltas: 100% (64/64), done.
(bluesky_2020_1) [sleepy ~/.ipython-bluesky] ll
total 12
drwxrwxr-x 2 29iduser s29user 4096 Feb 20 17:03 bin
drwxrwxr-x 3 29iduser s29user 4096 Feb 20 17:03 profile_bluesky
-rw-rw-r-- 1 29iduser s29user  195 Feb 20 17:03 README
(bluesky_2020_1) [sleepy ~/.ipython-bluesky] cp bin/bluesky29ID.sh ~/bin/
(bluesky_2020_1) [sleepy ~/.ipython-bluesky] bluesky
bluesky-0MQ-proxy        bluesky29ID.sh           bluesky_snapshot         bluesky_snapshot_viewer
(bluesky_2020_1) [sleepy ~/.ipython-bluesky] bluesky29ID.sh
Python 3.7.6 | packaged by conda-forge | (default, Jan  7 2020, 22:33:48)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.

IPython profile: bluesky
Activating auto-logging. Current session state plus future input saved.
Filename       : /home/beams22/29IDUSER/.ipython-bluesky/.logs/ipython_console.log
Mode           : rotate
Output logging : True
Raw input log  : False
Timestamping   : True
State          : active
I Thu-17:04:25 - ############################################################ startup
I Thu-17:04:25 - logging started
I Thu-17:04:25 - logging level = 10
I Thu-17:04:25 - /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/collection.py
I Thu-17:04:25 - /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/mpl/console.py
I Thu-17:04:28 - bluesky framework
I Thu-17:04:28 - /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/check_python.py
I Thu-17:04:29 - /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/initialize.py
Loading metadata history from /home/beams/29IDUSER/.config/bluesky/bluesky_history.db
/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/bluesky/utils.py:831: UserWarning: bluesky.utils.install_qt_kicker is no longer necessary and has no effect. Please remove your use of it. It may be removed in a future release of bluesky.
  warnings.warn("bluesky.utils.install_qt_kicker is no longer necessary and "
I Thu-17:04:35 - /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/user_dir.py
I Thu-17:04:35 - User code directory: /home/beams22/29IDUSER/.ipython-bluesky/bluesky_plans
I Thu-17:04:35 - /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/metadata.py
I Thu-17:04:36 - /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/callbacks.py
I Thu-17:04:37 - writing to SPEC file: /home/beams22/29IDUSER/.ipython-bluesky/20200220-170437.dat
I Thu-17:04:37 -    >>>>   Using default SPEC file name   <<<<
I Thu-17:04:37 -    file will be created when bluesky ends its next scan
I Thu-17:04:37 -    to change SPEC file, use command:   newSpecFile('title')
I Thu-17:04:37 - /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/aps_source.py

In [1]:

[prjemian]

Stunning error dump (which ultimately exited the bluesky session) when I tried to access the databroker component (mongodb database with archive of all previous data). Perhaps the server (groggy.xray.aps.anl.gov) has trouble? Is the mongodb server process running? The bluesky config file looks right.

[prjemian]

We can still test the ophyd config.

[prjemian]

Perhaps there is a different problem. Just tried a simple session. Autocomplete did not work. Another command crashed the session:

details

``` (bluesky_2020_1) [sleepy ~/.ipython-bluesky/profile_bluesky/startup] bluesky29ID.sh Python 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48) Type 'copyright', 'credits' or 'license' for more information IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help. IPython profile: bluesky Activating auto-logging. Current session state plus future input saved. Filename : /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/.logs/ipython_console.log Mode : rotate Output logging : True Raw input log : False Timestamping : True State : active I Thu-17:12:59 - ############################################################ startup I Thu-17:12:59 - logging started I Thu-17:12:59 - logging level = 10 I Thu-17:12:59 - /home/beams/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/collection.py I Thu-17:12:59 - /home/beams/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/mpl/console.py I Thu-17:12:59 - bluesky framework I Thu-17:12:59 - /home/beams/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/check_python.py I Thu-17:13:00 - /home/beams/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/initialize.py Loading metadata history from /home/beams/29IDUSER/.config/bluesky/bluesky_history.db /home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/bluesky/utils.py:831: UserWarning: bluesky.utils.install_qt_kicker is no longer necessary and has no effect. Please remove your use of it. It may be removed in a future release of bluesky. warnings.warn("bluesky.utils.install_qt_kicker is no longer necessary and " I Thu-17:13:03 - /home/beams/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/user_dir.py I Thu-17:13:03 - User code directory: /home/beams/29IDUSER/.ipython-bluesky/bluesky_plans I Thu-17:13:03 - /home/beams/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/metadata.py I Thu-17:13:03 - /home/beams/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/framework/callbacks.py I Thu-17:13:03 - writing to SPEC file: /home/beams22/29IDUSER/.ipython-bluesky/profile_bluesky/startup/20200220-171303.dat I Thu-17:13:03 - >>>> Using default SPEC file name <<<< I Thu-17:13:03 - file will be created when bluesky ends its next scan I Thu-17:13:03 - to change SPEC file, use command: newSpecFile('title') I Thu-17:13:03 - /home/beams/29IDUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/aps_source.py In [1]: show_ophyd_symbols() Traceback (most recent call last): File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/cache.py", line 109, in wrapper return dct[key] KeyError: ((), frozenset()) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/cache.py", line 109, in wrapper return dct[key] KeyError: ((), frozenset()) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/bin/ipython", line 11, in sys.exit(start_ipython()) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/__init__.py", line 126, in start_ipython return launch_new_instance(argv=argv, **kwargs) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/traitlets/config/application.py", line 664, in launch_instance app.start() File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/terminal/ipapp.py", line 356, in start self.shell.mainloop() File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 558, in mainloop self.interact() File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 541, in interact code = self.prompt_for_code() File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 469, in prompt_for_code **self._extra_prompt_options()) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/prompt_toolkit/shortcuts/prompt.py", line 997, in prompt return self.app.run() File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 810, in run return loop.run_until_complete(self.run_async(pre_run=pre_run)) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete return future.result() File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 777, in run_async return await _run_async2() File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 765, in _run_async2 await self.cancel_and_wait_for_background_tasks() File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 862, in cancel_and_wait_for_background_tasks await task File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/prompt_toolkit/buffer.py", line 1854, in new_coroutine await coroutine(*a, **kw) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/prompt_toolkit/buffer.py", line 1684, in async_completer document, complete_event File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/prompt_toolkit/completion/base.py", line 270, in get_completions_async document, complete_event File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/prompt_toolkit/completion/base.py", line 196, in get_completions_async for item in self.get_completions(document, complete_event): File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/terminal/ptutils.py", line 90, in get_completions yield from self._get_completions(body, offset, cursor_position, self.ipy_completer) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/terminal/ptutils.py", line 100, in _get_completions for c in completions: File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/core/completer.py", line 438, in _deduplicate_completions completions = list(completions) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/core/completer.py", line 1815, in completions for c in self._completions(text, offset, _timeout=self.jedi_compute_type_timeout/1000): File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/core/completer.py", line 1872, in _completions signature = _make_signature(jm) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/IPython/core/completer.py", line 989, in _make_signature return '(%s)'% ', '.join([f for f in (_formatparamchildren(p) for p in completion.params) if f]) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/cache.py", line 111, in wrapper result = method(self, *args, **kwargs) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/api/classes.py", line 428, in params for signature in self._get_signatures(): File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/api/classes.py", line 505, in _get_signatures return [sig for name in names for sig in name.infer().get_signatures()] File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/api/classes.py", line 505, in return [sig for name in names for sig in name.infer().get_signatures()] File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/cache.py", line 111, in wrapper result = method(self, *args, **kwargs) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/inference/compiled/mixed.py", line 129, in infer return _create(self._inference_state, compiled_value, module_context) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/inference/cache.py", line 43, in wrapper rv = function(obj, *args, **kwargs) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/inference/compiled/mixed.py", line 257, in _create result = _find_syntax_node_name(inference_state, python_object) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/inference/compiled/mixed.py", line 191, in _find_syntax_node_name module_node = _load_module(inference_state, path) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/inference/cache.py", line 43, in wrapper rv = function(obj, *args, **kwargs) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/inference/compiled/mixed.py", line 150, in _load_module cache_path=settings.cache_directory File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/inference/__init__.py", line 198, in parse return self.parse_and_get_code(*args, **kwargs)[0] File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/inference/__init__.py", line 195, in parse_and_get_code return grammar.parse(code=code, path=path, file_io=file_io, **kwargs), code File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/parso/grammar.py", line 78, in parse return self._parse(code=code, **kwargs) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/parso/grammar.py", line 153, in _parse cache_path=cache_path) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/parso/cache.py", line 174, in save_module _save_to_file_system(hashed_grammar, path, item, cache_path=cache_path) File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/parso/cache.py", line 178, in _save_to_file_system with open(_get_hashed_path(hashed_grammar, path, cache_path=cache_path), 'wb') as f: PermissionError: [Errno 13] Permission denied: '/home/beams/29IDUSER/.cache/jedi/CPython-37-33/17708d38681238dd1bba305ffeb4ac990f31754460f9a2481cbff62497eedbfb-83cca27a5e9be2ea8c0df61d39b4fb599d4b3e3a8da14b2af47b52a51934af34.pkl' If you suspect this is an IPython 7.12.0 bug, please report it at: https://github.com/ipython/ipython/issues or send an email to the mailing list at ipython-dev@python.org You can print a more detailed traceback right now with "%tb", or use "%debug" to interactively debug it. Extra-detailed tracebacks for bug-reporting purposes can be enabled via: %config Application.verbose_crash=True (bluesky_2020_1) [sleepy ~/.ipython-bluesky/profile_bluesky/startup] ```

[prjemian]

error starts with

In [1]: show_ophyd_symbols()
Traceback (most recent call last):
  File "/home/beams/29IDUSER/.conda/envs/bluesky_2020_1/lib/python3.7/site-packages/jedi/cache.py", line 109, in wrapper
    return dct[key]
KeyError: ((), frozenset())

The old JEDI problem for which we have a response commented out in the starter script (not everyone needs it).

[prjemian]

Works now (even autocomplete):

In [1]: show_ophyd_symbols()
========= ================= ======== ========
name      ophyd structure   EPICS PV label(s)
========= ================= ======== ========
aps       MyApsDevice
undulator MyUndulatorDevice ID29
========= ================= ======== ========

Out[1]: <pyRestTable.rest_table.Table at 0x7f42e8631690>

[prjemian]

info from the APS:

In [2]: device_read2table(aps)
========================================== ===================== ==========================
name                                       value                 timestamp
========================================== ===================== ==========================
aps_current                                102.00075796160932    2020-02-20 17:19:59.157790
aps_lifetime                               7.465751986709202     2020-02-20 17:19:55.729380
aps_aps_cycle                              2020-1                2020-02-20 17:18:27.711560
aps_machine_status                         USER OPERATIONS       2020-02-18 08:00:01.295432
aps_operating_mode                         Delivered Beam        2020-02-18 08:00:05.405082
aps_shutter_permit                         PERMIT                2020-02-18 08:00:05.404582
aps_fill_number                            6.0                   2020-02-18 08:00:05.404143
aps_orbit_correction                       0.0                   2020-02-20 17:18:26.963514
aps_global_feedback                        On                    2020-02-20 17:18:27.066070
aps_global_feedback_h                      On                    2020-02-20 17:18:27.066070
aps_global_feedback_v                      On                    2020-02-20 17:18:27.066070
aps_operator_messages_operators            Weyer, Orne           2020-02-20 15:08:26.444129
aps_operator_messages_floor_coordinator    Steve Bogdan (2-0101) 2020-02-20 07:54:45.824908
aps_operator_messages_fill_pattern         0+24x1 RHB            2020-02-15 05:58:52.657056
aps_operator_messages_last_problem_message                       2020-02-20 06:00:47.855662
aps_operator_messages_last_trip_message                          2020-02-20 06:00:52.240432
aps_operator_messages_message6             Top-Up is ongoing.    2020-02-20 06:00:43.838969
aps_operator_messages_message7                                   2020-02-16 05:11:10.273902
aps_operator_messages_message8                                   2020-02-16 05:11:06.707236
========================================== ===================== ==========================

Out[2]: <pyRestTable.rest_table.Table at 0x7f42e8623490>

[prjemian]

Now, the listruns() command produces an error that ends with:

ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

whic looks like the mongodb server is not running or does not accept our connection request.

[prjemian]

Sent request to IT for assist with the mongo. We can take data if we divert to use the mongodb server otz.xray.aps.anl.gov in my office. 3-ID is doing that now. We can do that until groggy/mongodb gets working again.

[prjemian]

Here's a picture of the default metadata we'll put into each scan's header (start document):

In [3]: print_RE_md()
RunEngine metadata dictionary:
======================== ==============================================
key                      value
======================== ==============================================
EPICS_BASE               /APSshare/epics/extensions-base/3.14.12.3-ext1
EPICS_BASE_PVT           /APSshare/epics/extensions-base/3.14.12.3-ext1
EPICS_CA_MAX_ARRAY_BYTES 800000
EPICS_EXTENSIONS         /APSshare/epics/extensions
EPICS_HOST_ARCH          linux-x86_64
beamline_id              APS 29-ID
login_id                 29iduser@sleepy.xray.aps.anl.gov
pid                      21615
proposal_id              testing
versions                 =========== ========
                         key         value
                         =========== ========
                         apstools    1.2.1
                         bluesky     1.6.0rc4
                         databroker  0.13.3
                         epics       3.4.1
                         matplotlib  3.1.3
                         numpy       1.18.1
                         ophyd       1.4.0rc4
                         pyRestTable 2020.0.2
                         spec2nexus  2021.1.7
                         =========== ========
======================== ==============================================

Out[3]: <pyRestTable.rest_table.Table at 0x7f66edace810>

[prjemian]

Here is the command to start a bluesky (ipython) session:

bluesky29ID.sh

[prjemian]

Fixed the mongodb configuration for bluesky and now databroker works:

In [1]: listruns()
========= ======================= ======= ======= ========================================
short_uid date/time               exit    scan_id command
========= ======================= ======= ======= ========================================
79ceba7   2017-10-06 11:19:28.153 success 64      count(num=300, detectors=['mono_ener ...
71d0043   2017-10-06 11:14:28.820 success 63      count(num=5, detectors=['mono_energy'])
fd79b99   2017-10-06 11:14:20.372 success 62      count(num=5, detectors=['mono_energy'])
a396400   2017-10-06 11:09:45.654 success 61      count(num=5, detectors=['aps_current'])
427c253   2017-10-06 11:09:29.449 success 60      count(num=1, detectors=['aps_current'])
e53f9a9   2017-10-06 11:09:11.673 success 59      count(num=1, detectors=['ca1_b'])
6bd2a1e   2017-10-06 11:06:59.282 success 58      count(num=1, detectors=['ca1_b'])
96a5b2a   2017-10-06 11:04:10.819 success 57      count(num=300, detectors=['D4'])
afbddf4   2017-10-06 11:03:25.731 success 56      count(num=300, detectors=['ca1_b'])
5bdbbdd   2017-10-06 11:02:16.371 success 55      count(num=300, detectors=['aps_curre ...
94b111d   2017-10-06 10:49:41.080 success 58      count(num=300, detectors=['aps_curre ...
f79a7f4   2017-10-06 10:48:21.996 success 57      count(num=300, detectors=['ca1_b'])
69dff04   2017-10-06 10:47:00.398 success 56      count(num=300, detectors=['ca1_b'])
c621155   2017-10-06 10:41:52.390 success 55      count(num=300, detectors=['D3', 'D4'])
07b9d29   2017-08-11 20:18:50.471         60      count(detectors=['EAV', 'TEY_C'], nu ...
0d1bc67   2017-08-11 16:06:32.676 success 59      count(detectors=['EAV', 'TEY_C'], nu ...
c681c32   2017-08-11 15:52:53.730 success 58      count(detectors=['EAV', 'TEY_C'], nu ...
8d0c6d9   2017-08-11 15:09:26.610 success 57      count(detectors=['EAV', 'TEY_C'], nu ...
1c90629   2017-08-11 15:03:52.756 success 56      count(detectors=['EAV', 'TEY_C'], nu ...
8c2217e   2017-08-11 15:00:47.456 success 55      count(detectors=['EAV', 'TEY_C'], nu ...
========= ======================= ======= ======= ========================================

Out[1]: <pyRestTable.rest_table.Table at 0x7f6909465a90>