notes on mxcube3 configuration

[spirit1317]

Hey. I started documenting a little bit how the configuration files work. You can see what I have already, I added this page:

https://github.com/spirit1317/mxcube3/wiki/Configuring-MXCube

you can download the file and add it to official wiki:

Configuring-MXCube.md

or, if you prefer I can make PR to README.md

[rhfogh]

Hi,

I do not know how to comment on the wiki, so an answer by mail.

Good to get this documented.

Two things one might add:

We also have .yml configuration for HardwareObjects, and in principle we have agreed that we are moving over to that - though it will take some time to get there.

There are some more requirements on the .yml side:

• YML-configured classes must subclass HardwareObjectYaml rather than HardwareObject

• You can add any property that you want in a .yml configutaiont file, but the relevant attribute must be defined in the class init; otherwise you get an error. This guarantees that all attributes added in configuration are visible in the code, to both programmers and linters.

If someone can tell me how to edit/add to the wiki I could propose an addendum.

Yours, Rasmus

On 28/10/2022 12:22, Mateusz Celary wrote:

Hey. I started documenting a little bit how the configuration files work. You can see what I have already, I added this page:

https://github.com/spirit1317/mxcube3/wiki/Configuring-MXCube https://github.com/spirit1317/mxcube3/wiki/Configuring-MXCube

you can download the file and add it to official wiki:

Configuring-MXCube.md https://github.com/mxcube/mxcube3/files/9887910/Configuring-MXCube.md

or, if you prefer I can make PR to README.md

— Reply to this email directly, view it on GitHub https://github.com/mxcube/mxcube3/issues/940, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK5QEEECSMYWDALAG3F4VDWFOZP3ANCNFSM6AAAAAARQ7GD4I. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[marcus-oscarsson]

Hi @spirit1317 and @rhfogh

Thanks @spirit1317 documentation efforts are always very appreciated and well needed :)

I think most of the documentation that you wrote above actually is for mxcubecore but it is of-course not less needed. What I would suggest is that we try to create the documentation from source and then add the bits that are missing. We have been using sphinx in the past (quite some time ago) but there are probably other tools today that should be investigated. The documentation build could then be added (as we discussed during our last code camp) to the CI.

This probably involves some work and there are some parts to improve as well, the first step would just be to generate the documentation from source. We have usually included some example configuration in the "header" of the HardwareObjects this could probably be included automatically.

We have discussed (and decided) to move move from XML to YAML configuration. However this work/switch needs to be done so that everybody is well informed about the consequences. Our hope is that it can be done relatively soon (during 2023).

The way we have envisioned this switch is to create a small tool that performs most of the work automatically. I was personally thinking that while we do this we can also create/generate pydantic configuration models like we do in mxcub3. In this way we can introduce a stricter validation of configuration files that also provides basic configuration and perhaps a logical place to store additional documentation for the corresponding hardware object.

[marcus-oscarsson]

With the above in mind. What I would suggest, if you feel motivated what you could try is to generate the documentation and see if corresponds to what you would like, we can then add the missing bits from what you just added.

We could perhaps give it a go with mkdocs instead of sphinx, what does the rest of you say ?

[oldfielj-ansto]

If we went with Sphinx for the MXCuBE3 and MXCuBE Core documentation, this is one area where I could help in getting it setup, as we have standardised on Sphinx for our other internal projects.

The main reason we've standardised on Sphinx is to make use of the Autodoc extension, which automatically parses Python code to generate documentation from the docstring and type hint annotations. This extension is only really useful for developer focused documentation and we host our beamline staff documentation separate to this Sphinx generated documentation.

One main difference which is worth considering here, is that MkDocs generates documentation from Markdown (".md") files, while Sphinx generates documentation from reStructuredText (".rst") files. If the main goal is to create a place to store information related to configuring and running MXCuBE, then MkDocs might be the better solution, as Markdown is a bit more approachable and most people can pick it up quite quickly.

I did find one extension for MkDocs, which seems to offer a similar functionality to the Sphinx Autodoc extension called MkAutodoc, but I haven't used it, so I'm not sure how fit for purpose it would be when it comes to auto-generated developer documentation.

As a side note, auto-generating documentation from the method, class and module docstrings generally requires that at a minimum we follow a standard docstring format, such as the Google Docstring Style or the NumPy Docstring Style. Most of the docstrings I have seen in both codebases do not follow either of these formats.

Reference Examples:

• FastAPI (MkDocs)
  • Source: https://github.com/tiangolo/fastapi/blob/master/docs/en/docs/index.md
  • Live: https://fastapi.tiangolo.com/index.html
• Bluesky (Sphinx)
  • Source: https://github.com/bluesky/bluesky/blob/master/docs/source/index.rst
  • Live: https://blueskyproject.io/bluesky/index.html

[marcus-oscarsson]

Thanks for the input, very valuable :)

I think that the autodoc feature is an absolute necessity, we used the one in sphinx a long time ago. At that time the doc strings used to be on "standard sphinx" format, we have since decided to move to the google doc strings. A lot of the doc strings should already be on the google doc-string format. We have given this link as an example in the contributing guidelines.

There is a MD format plugin for sphinx as well.

I'm open to either MKDocs or Sphinx assuming that autodoc works well in both. It would be nice if could use GitHub to host the documentation. I know that MD format is sported for this, but I'm not sure if it needs to be on some special structure or if other formats like RST are supported ?

[beteva]

Hi, thanks for all the details and the considerations. Certainly documentation is an important part of any software package and I do agree that autodoc feature is absolutely necessary. Also I think that each HardwareObject should include in its documentation an example of a configuration file.

Thanks @spirit1317 for trying to sort and document the configuration files. I see this as a good start , which deserves an effort to brush up and expand.

[oldfielj-ansto]

If it helps, we've already generated some documentation for the HardwareObjects in our facility specific code, we don't have it hosted yet, but as @marcus-oscarsson mentioned, it can be hosted via GitHub. I'll link the docs source below. https://github.com/AustralianSynchrotron/mxcubecore/tree/main/mxcubecore/HardwareObjects/ANSTO/docs/source

As for automating the documentation generation, while I don't have any GitHub actions written up for MXCuBE docs, I can provide another example AustralianSynchrotron/Sci-Comp-PyPi.

For context, this repository was setup when we moved our MXCuBE Core fork to GitHub, since we are unable to access our internal PyPi servers externally we needed a hosted solution and since we're cheap, we used GitHub pages to do this.

The way it works is simple enough, there is a GitHub actions workflow that comes with enabling GitHub pages on a branch, all you need to do in the settings is point it to the directory containing the generated "index.html" file and the action will deploy it automatically to that repositories GitHub.io link. e.g. https://australiansynchrotron.github.io/Sci-Comp-PyPi/

I'd imagine for the documentation we could setup a new branch, point GitHub pages at it and create a new GitHub action that would run each time a commit is merged to the "develop" branch. The action would run Sphinx/MkDocs to generate the HTML files, then push these new commits to the documentation specific branch.

[marcus-oscarsson]

Sounds great !, It was something like that I had in mind :). I'm in the middle of a few things right now but I'd be happy to give this a go together with PR 715 (or as much of 715 as we can reasonable apply right away). From there I think there are two things to do, look at how to apply the more stricter linting and whats missing in terms of documentation. If anyone is up for it, I'm happy if we give it a go :).

[marcus-oscarsson]

Closing as outdated