PyAEDT export the reStructuredText(.rst) files to JSON files for the Drupal migration.

[ampronovix]

Description of the feature

The goal of this task is to prepare the reStructuredText(.rst) content for the Drupal migration by exporting them into JSON files.

Steps for implementing the feature

The idea of this task is to expand the CI/CD pipeline(.github/workflows/ci-build.yml) in this repo to include the following steps:

1. Export the reStructuredText(.rst) files to JSON with the help of the Sphinx JSON builder:

   # Export the documentation files to JSON files.
   sphinx-build -M json doc/source doc-json -j auto -w build_errors.txt -N;

2. Flatten the generated nested files into a single directory:

   # Flattening a nested directory
   mkdir doc-flatten-json;
   # Move all the JSON file to the flatten directory.
   find . -name "*.fjson" -exec mv "{}" --backup=numbered ./doc-flatten-json \;
   # Make sure all the file has .json extensions instead of the .fjson
   for f in doc-flatten-json/* ; do mv -- "$f" "$f.json" ; done;
   # Move all static image files to the JSON flatten folder.
   mv doc-json/json/_images doc-flatten-json/_images;

3. Zip the flattened JSON directory:

   zip -r doc-flatten-json.zip doc-flatten-json;
   # Clean up build directories after the process is completed.
   rm -rf doc-json doc-flatten-json;

The above steps will generate a ZIP file containing all the JSON files that Drupal will use for the migration process, so Drupal will get the Zip file from this location:

https://github.com/pyansys/pyaedt/tree/main/doc-flatten-json.zip

All steps will run automatically as part of the repo CI/CD pipeline process and each time the documentation is exported to HTML should also build to JSON and generate the Zip file for Drupal with the fresh documentation content.

Useful links and references

1. Sphinx documentation generator tool.
2. Sphinx JSON HTML builder.

[germa89]

@ampronovix I'm not expert on this, but this part:

# Move all static image files to the JSON flatten folder.
mv doc-json/json/_images doc-flatten-json/_images;

Isn't going to generate problems for the images to be found? Since I presume the images are references in the json file with their paths. If you flat the structure, the corresponding image path should be also changed. Maybe I'm missing something.

[ampronovix]

Hi @germa89, Thanks for asking!

Drupal is going to take care of the process of transforming/resolving the relative/absolute paths from static images on the JSON files during the migration, so for this stage, it is only important to have the images included in the Zip so that when Drupal is resolving their references during the migration they can be found inside the Zip file.

Let me know if you have any other questions, looking forward to seeing this included in the CI/CD pipeline.

Thank you.

[MaxJPRey]

Hi @ampronovix,

After giving some thoughts about the process, I added some modifications. 1) I was able to generate the documentation as json files. 2) I was able to flatten the doc and zip it.

3) I did not add this zip file to the main branch in each of our repo (pyaedt, pymapdl...), which could create some confusion for some users by adding extra files not needed by the package. => Instead I created a new public repo where we could store all the zip file coming from all the repos: https://github.com/pyansys/devportal-doc-archive

In addition, instead of generating this file from the documentation built for every PR (dev-doc) we could use the doc generated for the release which is way more stable. Otherwise customers could get some info generated from unstable code. Would that work for you?

@ampronovix Please let me know if you can exploit the file pyaedt-doc-flatten-json.zip in the banch named drupal https://github.com/pyansys/devportal-doc-archive/tree/drupal

Adding @ChrisHarrold for visibility

[ampronovix]

Hi @MaxJPRey,

That's excellent news, I have tested the Drupal migration by pulling the PyAEDT Zip file from the location you indicated and this is working like a charm.

I am totally on board with the idea of putting all the Zip files from the dev-doc in the repository you created(https://github.com/pyansys/devportal-doc-archive) and using the doc generated for the release, this works :100: for us.

Thanks.

[ampronovix]

Hi @MaxJPRey

I noticed there was a recent deployment in the PyAEDT Documentation:

pyaedt-doc-flatten-json.zip Deploying to drupal from @ pyansys/pyaedt@d784ef4 5 hours ago

However, I see two problems:

1. The content of the new zip file was not generated properly, it contains JSON files with unexpected extensions that Drupal is not recognizing.
2. It looks like a dependency on Gtihub's static page generator was updated and that caused a problem as I now see a whole new section in the PyAEDT documentation that isn't linked to anything and wasn't there before: https://aedt.docs.pyansys.com/API/_autosummary/pyaedt.modules.CableModeling.Cable.add_cable_to_bundle.html

Can you please take a look and advise?

Thanks.

[MaxJPRey]

@ampronovix Sure I will. I just merged the pull request that generated the first version of the zip file. In addition, I am now using the release doc content and not the dev doc content because it will be more stable. (the PR is still pending for that). So basically, the content should be quite similar. I am going to check if we can get rid of the files with the unexpected extensions. I will also investigate the issue regarding the cable section in the documentation.

[ampronovix]

Hi @MaxJPRey,

We need to apply two small changes to the JSON Zip file generation:

1. We are missing two essential JSON files that were removed during the flattening process: globalcontext.json and searchindex.json

   Here is the updated flattening script(doc-zip-generator.sh) that we are using to make sure to include those files:

   #!/bin/bash
   # Flattening a nested directory, file: doc-zip-generator.sh
   mkdir -p doc-flatten-json;
   # Move all the JSON file to the flatten directory.
   JSON_FILES=$(find doc-json/json -type f -name "*.fjson" -o -name "*.json*");
   for filename in $JSON_FILES
   do
       x=${filename#./};
       y=${x// /-};
       mv -n ${x// /\ } "./doc-flatten-json/${y////-}";
   done
   # Move all static image files to the JSON flatten folder.
   mv -n doc-json/json/_images doc-flatten-json/_images;
   # Zip the flattened JSON directory:
   zip -r doc-flatten-json.zip doc-flatten-json;
   # Clean up build directories after the process is completed.
   rm -rf doc-json doc-flatten-json;

   We simply run ./doc-zip-generator.sh in the root of the project.

2. About the version control tagging in the Zip, We noticed this in the sphinx-build documentation:

   globalcontext.json: The filename for the file that contains the “global context”. This is a dict with some general configuration values such as the name of the project.

   This file that is included in the JSON generation contains metadata information about the project in general, and you can set the release and version by using the command option via -D setting=value, for example:

   sphinx-build -M json doc/source doc-json -j auto -D release=0.6.21 -D version=0.6.21;

   This way the version and release will be passed to the globalcontext.json file:

   

   Can you dynamically pass those two variables when running the JSON build so that each zip file contains the version and release tag they belong to?

Let me know if you have any questions, Thanks.

[MaxJPRey]

Hi @ampronovix, I am working on the first point. It should be merged soon. Regarding the second point, I don't see any limitation. I will do it too.

[jorgepiloto]

Pinging here the @pyansys/pyansys-core team.

We need to get familiar with the JSON doc rendering pipeline. Ideally, we should add these in pyansys/actions to minimize the impact in our projects. We already support JSON builders, but no flattening logic is provided yet. Note that pyansys/actions has a scripts directory in case we need to host a more complex logic.

[MaxJPRey]

@jorgepiloto this is the goal. It will be added in the pyansys/actions eventually and deployed in the different products. Here it is just the first step to get the script up and running and make sure the content can be used in the dev-doc portal and display something consistent and useful.

[ampronovix]

Hi @MaxJPRey, thanks for applying the requested changes.

I have two more questions/requests:

1. Can you help me coordinate the generation of JSON zip files for the other 10 packages?
2. Who should be briefed on the other teams to implement this JSON build integration into the other 8 packages?

Let me know if you need anything from my end to get this going. Ideally Drupal will try to fetch the Zip files from the following URLs for the other packages:

Package	Expected JSON zip URL
PyDPF-Core	https://github.com/pyansys/devportal-doc-archive/blob/drupal/pydpfcore-doc-flatten-json.zip?raw=true
PyDPF-Post	https://github.com/pyansys/devportal-doc-archive/blob/drupal/pydpfpost-doc-flatten-json.zip?raw=true
PyMAPDL	https://github.com/pyansys/devportal-doc-archive/blob/drupal/pymapdl-doc-flatten-json.zip?raw=true
PyMAPDL Reader	https://github.com/pyansys/devportal-doc-archive/blob/drupal/pymapdlreader-doc-flatten-json.zip?raw=true
PyFluent	https://github.com/pyansys/devportal-doc-archive/blob/drupal/pyfluent-doc-flatten-json.zip?raw=true
PyFluent-Parametric	https://github.com/pyansys/devportal-doc-archive/blob/drupal/pyfluentparametric-doc-flatten-json.zip?raw=true
PyFluent-Visualization	https://github.com/pyansys/devportal-doc-archive/blob/drupal/pyfluentvisualization-doc-flatten-json.zip?raw=true
PyPIM	https://github.com/pyansys/devportal-doc-archive/blob/drupal/pypim-doc-flatten-json.zip?raw=true
Granta MI BoM Analytics	https://github.com/pyansys/devportal-doc-archive/blob/drupal/grantami-doc-flatten-json.zip?raw=true
Shared Components	https://github.com/pyansys/devportal-doc-archive/blob/drupal/shared-doc-flatten-json.zip?raw=true

cc @jorgepiloto

Thanks.

[MaxJPRey]

Hi @ampronovix,

We are going to create an action in pyansys/actions based on the code integrated in the PyAEDT workflow. I will sync with @jorgepiloto to create it.

As soon as it is done, we will talk internally to the other teams (the one in charge of the repositories you mentioned above) to make sure that this action is embedded into in their CI/CD.

Thanks for your feedback.

[MaxJPRey]

Adding @anspcoinaud for visibility.

[ChristophWeber]

Hi @MaxJPRey, we noticed inconsistent data in the current PyAEDT repo zip file that creates inconsistent version numbers of the developer portal. We think it may simply be a consequence of your rolling out a dev version for us as proof of concept and will resolve itself when you produce a proper new version. Can you confirm?

[MaxJPRey]

Hi @ChristophWeber Yes, exactly. It should be a released version and consistent across the entire doc. We will investigate and correct this as soon as possible. Thanks.