movedo / MoVeDo

build-system for Modular, Versioned Documentation
GNU General Public License v3.0
6 stars 3 forks source link

Modular system for tools management for OSH documentation #8

Open hoijui opened 4 years ago

hoijui commented 4 years ago

hello all of you! I am working on something that is somehow related to this. though it is more about hardware documentation, the question is a more general one, about an architecture for .. tools/plugins/scripts that are useful in OSH documentation repos.

Introduction through an example

So... let's imagine we are documenting a machine, that is made up of some mechanical parts and some electronics. The mechanical parts are designed in FreeCAD (FC), the electronic ones in KiCad (KC). So our repo contains some Markdown files, some FC files and some KC files. In our CI/CD (build-bot) job, we want to generate renders in SVG and/or PNG for the FC and KC files, and STL files from the FC files, and Gerbers from the KC files (that last one I am working on/have mostly done now). Because FC uses a binary format, we also want to pre-process these files with a git filter before storing them in the repo (for example with ReZipDoc).

MoVeDo (pronunciation is like an Italian person would say it, btw.) handles the Markdown part, ReZipDoc the FC storage issue, what I am working on now the Gerbers for KC, and the rest is still to be done, and we need more tools for FC and KC, but probably also for other formats not present in this example (OpenSCAD would be a candidate).

The Question

How to best organize this all?

I imagine a kind of flexible system of small tools, that can also be replaced one by one, if desired, and are easy to modify.

My personal wishes for the requirements would be:

... but I am open for a completely different idea as well. It would be especially cool, if a system that fits in here would already exist of course.

Any ideas/inspirations/... ?

hoijui commented 4 years ago

A sample OSH docu project could look like this:

build/docu/html/...
build/docu/md/...
build/docu/pdf/...
build/parts/case/model-tech-drawing.svg
build/parts/case/model-render.png
build/parts/electronics/circuit.png
build/parts/electronics/circuit.svg
build/parts/electronics/circuit-gerbers.zip
doc-tools/docu-compiler # git sub-module pointing at MoVeDo
doc-tools/free-cad-storage # git sub-module pointing at ReZipDoc
doc-tools/ki-cad-ploter # git sub-module pointing at a yet-to-be-created repo
doc-tools/manager # git sub-module pointing at a yet-to-be-created repo for doc-tools management
parts/case/docu.md
parts/case/model.fcstd
parts/electronics/docu.md
parts/electronics/circuit.pro
parts/electronics/circuit.kicad_pcb
parts/electronics/circuit.net
parts/electronics/circuit.sch
docu.md
LICENCE.md
README.md

or in tree style:

├── build
│   ├── docu
│   │   ├── html
│   │   ├── md
│   │   └── pdf
│   └── parts
│       ├── case
│       │   ├── model-render.png
│       │   └── model-tech-drawing.svg
│       └── electronics
│           ├── circuit-gerbers.zip
│           ├── circuit.png
│           └── circuit.svg
├── doc-tools
│   ├── docu-compiler
│   ├── free-cad-storage
│   ├── ki-cad-ploter
│   └── manager
├── parts
│   ├── case
│   │   ├── docu.md
│   │   └── model.fcstd
│   └── electronics
│       ├── circuit.kicad_pcb
│       ├── circuit.net
│       ├── circuit.pro
│       └── circuit.sch
├── LICENCE.md
└── README.md
kasbah commented 4 years ago

This issue/discussion surprises me somewhat as I thought this is a tool for standards writing and publishing rather than for hardware documentation. If the development focus branches out like this I worry that the standards writing use-case could be left behind and the tool will become more complicated and harder to use. It gives me second thoughts whether we should try use MoVeDo for the OKH standard going forward.

Nevertheless, some possible prior art and inspiration for the hardware documentation use-case could be:

  1. My project: kitspace.org, specifically the kitspace YAML manifest which was also part of the the inspiration for the OKH manifest standard. In OKH we try and apply this approach to all hardware designs, not just electronics. The main difference to what you are describing is rather than impose a pre-defined structure on the user, let them use their own structure but add paths to the manifest so the tools know where to find things. In the kitspace.yaml we already have:

    • gerbers: a path to a gerbers folder
    • bom: a path to a BOM in csv, tsv, ods or excel formats with pre-defined possible headings
    • eda: a field with sub-fields
      • type: currently makes use of kicad or eagle
      • pcb: a path to the .kicad_pcb or .brd file
  2. GitBuilding is a tool created by @julianstirling and it's very focused on the writing side, counting parts and collating the bill of materials. It produces nice documentation outputs for web and PDF. For the webview we have already integrated a preview view of STL and some other 3d formats. Maybe @julianstirling can say more about this.

hoijui commented 4 years ago

thank you for the feedback @kasbah , again! :-)

sorry for the confusion! The issue is conceptually rather in a super-space of MoVeDo, not part of MoVeDo. I just opened the discussion here, because I did not know where else to put it. MoVeDo might be used as a module in that system (in its current conceptualization), but it should will be usable stand-alone as well.

I see MoVeDo as a general document management tool, not limited to standards, though. Is there anything specific to standards, that is not in other documentations?

Regarding your two suggestions:

  1. I did not think of meta-data files for specifying locations... nice! :-) We could use the best of both, as does Maven or Gradle with the Convention over configuration concept; meaning:

    If not further specified (in a meta-data file) the default structure can be assumed.

    This has the benefit of project owners having to configure less, and it will let OSH projects gravitate towards a common, optimal structure. Such a structure is necessarily usually more complex then what one would naturally use for a very simple project, but comes with many benefits for automatic processing of project, especially in a multi-tool environment, which is what I would like to encourage.

  2. I had a look at GitBuilding, and it does not fulfill many of the items from my wish-list from above:

    • [ ] to prevent lock-ins - our whole system or parts of it should be replaceable with alternatives
    • [ ] to make it modular
    • [x] (being worked on) to keep common functionality in a kind of library, which could be yet an other tool
    • [x] platform independent - if not, it has to be Linux compatible at the very least as it will mostly run on CI jobs
    • [ ] multi-programming-language support - each tool could use a different language
    • [ ] plumbing through CLI interfaces
    • [ ] (does not apply because no sub-modules) to have a kind of comfortable managing interface to list, update, add and remove tools in any OSH documentation repo
    • tools being git repos, and added to the OSH doc repo as git sub-modules

It also introduces a new document format -> vendor-lock-in & bad for multi-tool support. This document format mixes structure information (BOM) within unstructured data (Markdown like text), which I think is not a good idea.

I have no good concept of how BOMs should be handled yet myself, but I imagine them being generated from different, structured parts of the project data:

These different sources of BOM would all end up in one common, simple, text-based file, maybe CSV. From there we can again include it in Markdown documents, generate HTML or PDF views, reference parts of it, filter out just the electronics, or something else.

hoijui commented 4 years ago

while I have multi-programming-language support on my wish-list, and I think it should stay there, there should maybe be a best-practice suggestion though. As of now, MoVeDo for example, is half BASH half Python. I personally do not like Python very much, but ... I should maybe at least try to write one CLI tool in it, and later maybe start rewriting part by part of MoVeDo's BASH scripts int Python.

Would anyone have a better suggestion then Python? Maybe Julia?

Python (vs BASH) for this scenario (brackets mean: low importance):

Python compared to possible other languages:

I guess... it is optimal, right? (just trying to convince myself against my dislike of it ;-) )

hoijui commented 4 years ago

This document format mixes structure information (BOM) within unstructured data (Markdown like text), which I think is not a good idea.

... though, if the part of GitBuilding that parses the document and extracts the BOM data could be used separately, even such a document could be made to feed its data into the common BOM file.

hoijui commented 4 years ago

in my current concept, the build and the doc-tools dirs (and everything below them) are specified by convention, but the structure of the content (most of which I put into the parts dir in the example) is still totally free. It simply is mirrored in build/gen-src at build-time.

That is roughly also how MoVoDo does it.

julianstirling commented 4 years ago

The idea that we write the BOM in a separate file as a CSV is exactly the problem we are trying to solve. Having CSV files for your BOM stops you making you documentation modular and leads to documentation and BOM getting out of sync. Instead we embed the information about the parts used in the text for each page, this allows us to create total bills of materials and export the as CSV (this export has been merged to master but not released). Probably the best explanation of the structure is here

hoijui commented 4 years ago

If there is already BOM information in other tools (say FreeCAD or KiCad), then having it in your Markdown files is still duplication, even if you do not put it (manually) into a CSV file. Thus I assume, your use-case is for projects that do not have any such things like mechanical-, electronics-, textile- or other model files, right?

When having it in a CSV file, one can generate markdown from that, and link to it, or include it in a generated output file in the same document.

I agree though, that in the absence of any other model containing BOM information, using your system might be a viable - or even optimal - solution, if it works nicely with tools that do not know about your special syntax. An example could be simple projects without electronics and 3D models. It could be one source of BOM data, and as you explain it, it sounds like it would fit right into such a scenario, especially with your new developments.


julianstirling commented 4 years ago

If there is already BOM information in other tools (say FreeCAD or KiCad), then having it in your Markdown files is still duplication, even if you do not put it (manually) into a CSV file. Thus I assume, your use-case is for projects that do not have any such things like mechanical-, electronics-, textile- or other model files, right?

Yes and no. The thing is that even if the information is already in the CAD you are still going to be referring to the parts by name in the text.

Example

I have a FreeCAD assembly with 7 screws. I have assembly instructions which explain the toque to tighten these 7 screws. If I modify the assembly in FreeCAD and then export the BOM then the BOM (whether it is in CSV, Markdown, XML, YAML, whatever) is out of date with the text.

However, if I have tagged in the text:

Take seven of the [m3x16 hexagon cap screws](hexscrewlink.md){Qty:7} and tighten them into the following locations, to a torque of 1.3 Nm.

Now if can write a script that checks the BOM of the process I have described in my assembly instructions matches the BOM of that is in the FreeCAD file. As far as I can tell this is necessary because the software cannot competently adjust the assembly instructions. A human needs to go through when things change and check that it still make sense.



Taking a step back. Do you really feel that having one program that automates PDFs for Standards, assembly docs, user manuals is a good thing? I would say that the formatting needed fro a PDF of a standard is very formal sectioned text. For assembly docs you want to move towards something akin to big picture, minimal text side by side. Of course a single program could do all of this but it seems to be straying from the Unix principle you are championing of doing one thing and doing it well.

For OKH we are interested in trying MoVeDo for the standard. as this is what it can do now. If in the future we want to join efforts to make an all singing all dancing collection of tools for OSH documentation, then we should take a step back. The only sensible way to do this is for us to:

Working together to set our objectives is far more productive than one person defining a personal manifesto of requirements and berating anyone that hasn't already lived up to their arbitrary standards.


I suggest that we close this issue and that we move the discussion of wider documentation tools and formats Hardware Documentation Consensus Groups' Tool Comparison Repo. We will also be having a call this Wednesday if you would like to join.

hoijui commented 4 years ago

... The thing is that even if the information is already in the CAD you are still going to be referring to the parts by name in the text.

Ahhhh.. click ... hmm :/ yeah, I did totally not think of that, thank you for taking the time to explain!

Example

I have a FreeCAD assembly with 7 screws. I have assembly instructions which explain the toque to tighten these 7 screws. If I modify the assembly in FreeCAD and then export the BOM then the BOM (whether it is in CSV, Markdown, XML, YAML, whatever) is out of date with the text.

However, if I have tagged in the text:

Take seven of the [m3x16 hexagon cap screws](hexscrewlink.md){Qty:7} and tighten them into the following locations, to a torque of 1.3 Nm.

Now if can write a script that checks the BOM of the process I have described in my assembly instructions matches the BOM of that is in the FreeCAD file. As far as I can tell this is necessary because the software cannot competently adjust the assembly instructions. A human needs to go through when things change and check that it still make sense.

So let me see if I got it right:

I change my FreeCAD model from something with 7 to sometihng with 8 screws, and I forget to adjust the text/docu. The CI build extracts BOM info from both the FreeCAD model and the docu (in your format), and sees that one has 8 screws, and the other 7, and then generates a warning/error, so a human can make a fixing commit?

* **vendor lock-in** - The logical conclusion of your argument is that ANY new file format is lock in. Creating markdown was lock in, creating ODT was lock-in. Anything that was not a ASCII encoded plain text file is lock in. The unfortunate reality of the real world is that if no format exists that supports what you want to do you have to create one. From day 1 GitBuilding has had plain markdown as an output because it was written as a script that converted the meta data links in place. This means you can always get your data out, and we also checked from day 1 that the documentation was readable in GitLab/GitHub even if not perfect. The idea that we are trying "minor lock-ins" is absurd, offensive. If the formats that exist cannot support the needs of the open hardware community, we need to write our own. We could start from scratch, making it hard to replicate, or we could supplement a widely used standard.

I guess, lock-in fits better then vendor lock-in, but even that is not optimal. either way, it is no offense, and nothig to be offended by. You seem to ignore what I said about this before. It is not about an ill-intention that I am accusing you of, but about a paractical situation that arises out of your design choices. Your choices might be the best/optimal ones in the scenario, but they are not perfect in every way. maybe, as you say, creating a new format is unavoidable or optimal, but making a new format has downsides, even if you also minimize them, with the tool that separates the BOM info and the pure markdown into separate files. I am quite convinced by now though, that it is pprobably optimal.

* **modularity** - The unix BASH philosophy is wonderful. Yet unfortunately if you want a simple to install cross platform app, that is easy to debug and unit test you cannot do it a set of small bash scripts. I have learned this the hard way. The codebase is a python library. We are splitting out the core functionality from the more UI side of the project. The library functions are very fine grained small functions that try to adhere to the KISS principle. And being python libraries you can call them from other code. One of the main goals it to make software that can be used by everyone, not just nerds like us that use linux. So rag-tag-ball of BASH scripts was not going to cut it.

It is not the UNIX BASH philosophy, it is the UNIX philosophy. although BASH is the most common tool that does the plumbing, the important thing is the interface the tools support. any shell can do that plumbing, alos one implemented in python, or something not a shell can do it. the interface is basically: stdin, stdout, stderr, command line arguments, and a general gravitation towards text-based data, new-lines or '\0' separated units of text, with tabs or spaces as sub-dividers, ... and that's basically it.\ If you define these tools as the units, then you can also do unit-testing on them. Using this interface, we also get multi-language support with it.

Maybe though, an other interface would make sense for us, but I don't know one. I guess this would be IPC, which I only know by name, and seems to be a very general concept... but stuff reffering to itsself as IPC might be in the ballpark.

I am from the Java world, where I liked OSGi a lot, which I think is similar to the Spring framework Core Technologies, but that is of course Java only.

The "UNIX interface" is kind of very loose.. one can not define an abstract interface with meta-data or something like that, which is possible in OSGi, and I think in some functional languages or frameworks, there is evne better stuff.

Any replacement for the UNIX thing though, will be less widely known and more complicated to use for most people, I think.

* **multi-language support** - If you think of GitBuilding/BuildUp as the step that takes you from markdown-with-part-metadata, to markdown or html etc. And then other tools as the ones that say create the STLs, publish, etc, etc. Then you can use what ever languages.

cool, we are getting to mor and more common understanding! :-)

* **managing interface** - Assuming you have python 3.7+ you can run `pip install gitbuilding` on Windows/Mac/Linux to install the current stable version. As there are some external dependencies (especially on windows), and a lot of windows users don't like command line there is a plan to bundle the software up into stand alone downloadable packages. This is as well as, not instead of the ability to `pip install` as this is how I want to install it.

cool! :-)

* **git sub-module** - We can agree to disagree on whether this is a good thing. But yes, seeing as GitBuilding is in a git repo you could do this if you so desired.

+1

Taking a step back. Do you really feel that having one program that automates PDFs for Standards, assembly docs, user manuals is a good thing? I would say that the formatting needed fro a PDF of a standard is very formal sectioned text. For assembly docs you want to move towards something akin to big picture, minimal text side by side. Of course a single program could do all of this but it seems to be straying from the Unix principle you are championing of doing one thing and doing it well.

I never though of it like a thing specifically geared towards specfication/norm documentation. I actually used it from the start, and still do, also for a recipee blog. That is a good aproach I think, because it automatically makes me implement eveerything in a way that is compatible for many things. You should also keep in mind, that MoVeDo does not do any of the grunt work, but uses tools for all of that, and these tools themselfs are quite powerfull, and all of them are geared towards general documentaiton. For example, right now we are in the proces sto improve the title-page of the DIN SPEC 3015. This will be done by allowing to supply a custom pandoc template oand/or live-patching the default one, if that proves feasible.

So there is really nothing that is norm-specific in MoVeDo as of now.

For OKH we are interested in trying MoVeDo for the standard. as this is what it can do now. If in the future we want to join efforts to make an all singing all dancing collection of tools for OSH documentation, then we should take a step back. The only sensible way to do this is for us to:

* use and understand each others current projects

* agree not to accuse each other of unethical practices such as lock-in if we disagree on how something is implemented (i.e suggest a specific and viable alternative rather than finger pointing)

* each make a list of ideal requirements

* work together to align these requirements to a single, group agreed, set of requirements.

sounds good!

Working together to set our objectives is far more productive than one person defining a personal manifesto of requirements and berating anyone that hasn't already lived up to their arbitrary standards.

I opened a discussion here, and supplied my wish-list, explicitly stating it as "my personal wish-list", suggesting that it is not god's unquestionable list; I guess that was not clear enough then, sorry. the list is not arbitrary, but comes from my experience. You are free to extend/adjust the list in this discussion, and I would like you to, as you seem well capable to do so.

I suggest that we close this issue and that we move the discussion of wider documentation tools and formats Hardware Documentation Consensus Groups' Tool Comparison Repo. We will also be having a call this Wednesday if you would like to join.

I am in favor of moving it, but would prefer for people not to require an IEEE account to take part. Do you have something for that as well? It recently took me 4 days to get an account there. maybe it was just the wrong time of the year, but also if not.. it is a barrier.

Was the call yesterday or is it next week?

julianstirling commented 4 years ago

I change my FreeCAD model from something with 7 to something with 8 screws, and I forget to adjust the text/docu. The CI build extracts BOM info from both the FreeCAD model and the docu (in your format), and sees that one has 8 screws, and the other 7, and then generates a warning/error, so a human can make a fixing commit?

That is a use case I have had in mind. As FreeCAD has no default assembly workbench this is not something we are implementing in the near term. But we are making machine readable BOMs from the output so that you can do things like consistency check between CAD and docs. I am talking to a friend who uses Solidworks to generate CSV files from assemblies, for a proof of principle. Of course Solidworks will never run on the CI unless he has his own local windows runner, but for his use case this is less important.

I opened a discussion here, and supplied my wish-list, explicitly stating it as "my personal wish-list"

My fault, I got grumpy. Sorry

but would prefer for people not to require an IEEE account to take part

My preference would have been standard GitLab. I think Tobey's plan is to try to tap into some of the standards experience of IEEE, and also to trial the IEEE platform that seems to be taking open source hardware seriously. I think at first we should raise the problem of slow account creation as a barrier with IEEE, if it becomes a problem we can migrate.

Was the call yesterday or is it next week?

The call is next week. We are a bit disorganised arranging it, I will makes sure I let you know.