Closed ampronovix closed 1 year ago
@ampronovix I'm not expert on this, but this part:
# Move all static image files to the JSON flatten folder.
mv doc-json/json/_images doc-flatten-json/_images;
Isn't going to generate problems for the images to be found? Since I presume the images are references in the json file with their paths. If you flat the structure, the corresponding image path should be also changed. Maybe I'm missing something.
Hi @germa89, Thanks for asking!
Drupal is going to take care of the process of transforming/resolving the relative/absolute paths from static images on the JSON files during the migration, so for this stage, it is only important to have the images included in the Zip so that when Drupal is resolving their references during the migration they can be found inside the Zip file.
Let me know if you have any other questions, looking forward to seeing this included in the CI/CD pipeline.
Thank you.
Hi @ampronovix,
After giving some thoughts about the process, I added some modifications. 1) I was able to generate the documentation as json files. 2) I was able to flatten the doc and zip it.
3) I did not add this zip file to the main branch in each of our repo (pyaedt, pymapdl...), which could create some confusion for some users by adding extra files not needed by the package. => Instead I created a new public repo where we could store all the zip file coming from all the repos: https://github.com/pyansys/devportal-doc-archive
In addition, instead of generating this file from the documentation built for every PR (dev-doc) we could use the doc generated for the release which is way more stable. Otherwise customers could get some info generated from unstable code. Would that work for you?
@ampronovix Please let me know if you can exploit the file pyaedt-doc-flatten-json.zip
in the banch named drupal
https://github.com/pyansys/devportal-doc-archive/tree/drupal
Adding @ChrisHarrold for visibility
Hi @MaxJPRey,
That's excellent news, I have tested the Drupal migration by pulling the PyAEDT Zip file from the location you indicated and this is working like a charm.
I am totally on board with the idea of putting all the Zip files from the dev-doc in the repository you created(https://github.com/pyansys/devportal-doc-archive) and using the doc generated for the release, this works :100: for us.
Thanks.
Hi @MaxJPRey
I noticed there was a recent deployment in the PyAEDT Documentation:
pyaedt-doc-flatten-json.zip Deploying to drupal from @ pyansys/pyaedt@d784ef4 5 hours ago
However, I see two problems:
Can you please take a look and advise?
Thanks.
@ampronovix Sure I will.
I just merged the pull request that generated the first version of the zip file.
In addition, I am now using the release doc content and not the dev doc content because it will be more stable. (the PR is still pending for that).
So basically, the content should be quite similar. I am going to check if we can get rid of the files with the unexpected extensions.
I will also investigate the issue regarding the cable
section in the documentation.
Hi @MaxJPRey,
We need to apply two small changes to the JSON Zip file generation:
We are missing two essential JSON files that were removed during the flattening process: globalcontext.json and searchindex.json
Here is the updated flattening script(doc-zip-generator.sh
) that we are using to make sure to include those files:
#!/bin/bash
# Flattening a nested directory, file: doc-zip-generator.sh
mkdir -p doc-flatten-json;
# Move all the JSON file to the flatten directory.
JSON_FILES=$(find doc-json/json -type f -name "*.fjson" -o -name "*.json*");
for filename in $JSON_FILES
do
x=${filename#./};
y=${x// /-};
mv -n ${x// /\ } "./doc-flatten-json/${y////-}";
done
# Move all static image files to the JSON flatten folder.
mv -n doc-json/json/_images doc-flatten-json/_images;
# Zip the flattened JSON directory:
zip -r doc-flatten-json.zip doc-flatten-json;
# Clean up build directories after the process is completed.
rm -rf doc-json doc-flatten-json;
We simply run ./doc-zip-generator.sh
in the root of the project.
About the version control tagging in the Zip, We noticed this in the sphinx-build documentation:
globalcontext.json: The filename for the file that contains the “global context”. This is a dict with some general configuration values such as the name of the project.
This file that is included in the JSON generation contains metadata information about the project in general, and you can set the release and version by using the command option via -D setting=value
, for example:
sphinx-build -M json doc/source doc-json -j auto -D release=0.6.21 -D version=0.6.21;
This way the version and release will be passed to the globalcontext.json
file:
Can you dynamically pass those two variables when running the JSON build so that each zip file contains the version and release tag they belong to?
Let me know if you have any questions, Thanks.
Hi @ampronovix, I am working on the first point. It should be merged soon. Regarding the second point, I don't see any limitation. I will do it too.
Pinging here the @pyansys/pyansys-core team.
We need to get familiar with the JSON doc rendering pipeline. Ideally, we should add these in pyansys/actions to minimize the impact in our projects. We already support JSON builders, but no flattening logic is provided yet. Note that pyansys/actions has a scripts directory in case we need to host a more complex logic.
@jorgepiloto this is the goal. It will be added in the pyansys/actions eventually and deployed in the different products. Here it is just the first step to get the script up and running and make sure the content can be used in the dev-doc portal and display something consistent and useful.
Hi @MaxJPRey, thanks for applying the requested changes.
I have two more questions/requests:
Let me know if you need anything from my end to get this going. Ideally Drupal will try to fetch the Zip files from the following URLs for the other packages:
cc @jorgepiloto
Thanks.
Hi @ampronovix,
We are going to create an action in pyansys/actions based on the code integrated in the PyAEDT workflow. I will sync with @jorgepiloto to create it.
As soon as it is done, we will talk internally to the other teams (the one in charge of the repositories you mentioned above) to make sure that this action is embedded into in their CI/CD.
Thanks for your feedback.
Adding @anspcoinaud for visibility.
Hi @MaxJPRey, we noticed inconsistent data in the current PyAEDT repo zip file that creates inconsistent version numbers of the developer portal. We think it may simply be a consequence of your rolling out a dev version for us as proof of concept and will resolve itself when you produce a proper new version. Can you confirm?
Hi @ChristophWeber Yes, exactly. It should be a released version and consistent across the entire doc. We will investigate and correct this as soon as possible. Thanks.
Description of the feature
The goal of this task is to prepare the reStructuredText(.rst) content for the Drupal migration by exporting them into JSON files.
Steps for implementing the feature
The idea of this task is to expand the CI/CD pipeline(.github/workflows/ci-build.yml) in this repo to include the following steps:
Export the reStructuredText(.rst) files to JSON with the help of the Sphinx JSON builder:
Flatten the generated nested files into a single directory:
Zip the flattened JSON directory:
The above steps will generate a ZIP file containing all the JSON files that Drupal will use for the migration process, so Drupal will get the Zip file from this location:
All steps will run automatically as part of the repo CI/CD pipeline process and each time the documentation is exported to HTML should also build to JSON and generate the Zip file for Drupal with the fresh documentation content.
Useful links and references