💥 Big Changes Ahead! :boom:

[tajmone]

These are the big changes we'll be working on next:

• [x] Switch to Rouge syntax highlighter (#130)
  • More powerful syntax coverage for ALAN.
  • Callouts supported.
  • Usable for both HTML5 and PDF backends (asciidoctor-pdf only).
  • No more highlight dependency, and since Rouge is a Ruby gem, no additional languages dependencies either (just the Rouge Gem).
• [x] UTF-8 Encoding for ALAN (#131)
  • Now ALAN Beta8 supports encoding ALAN sources and solutions in UTF-8, and automatically generated UTF-8 encoded transcripts.
  • Editing ALAN sources in this project will be easier for contributors, since UTF-8 is the de facto standard in modern editors, whereas preserving ISO-8859-1 integrity was hard.
  • Our ALAN sources and script will be easier to manipulate via third party tools that don't support ISO encoding.
• [x] Adopting Rake (#132)
  • We'll be dropping shell and batch scripts in out toolchain and handle all build tasks via Rake — Ruby's Make — which will make our toolchain fully cross platform (usable in Bash, CMD, PowerShell, etc.).
    • Initially we'll have to use Rake from Bash, since some tasks depend on specific Unix tools which are not available under Win CMD, but we'll eventually replace them with pure Ruby code, making the toolchain fully platform- and terminal-agnostic.
  • Rake builds are much faster because Rake only updates files if their dependencies have changed.
  • Rake is powerful because Rakefiles are Ruby files, so you can add custom code to any task. Rake is better than Make, it doesn't have all the quirks and problems of Make, and natively does much more than what Make can.
  • We've started developing on ALAN i18n reusable Ruby/Rake modules specific for ALAN and Asciidoctor projects, offering methods that only need a folder path to programmatically generated the required tasks on the fly, so we won't need to tweak the Rakefile when files are added or deleted.
  • Rake is easy to use, you only need to type rake in any folder inside the repository, not just the root. Using namespaces, we can invoke specific tasks (e.g. rake docs:man to build just the manual, etc.).
  • Rake is now part of the Ruby language so, since this project already need Ruby for Asciidoctor, we won't be adding any dependencies.

Since these involve significant changes to the whole repository, we won't be tending other pending tasks until all the above steps have become effective.

[tajmone]

Work Will Resume Soon...

@AnssiR66, the reason I haven't been updating the StdLib in all these months was due to upcoming changes of ALAN adopting UTF-8, the switch to Rouge, and the adoption of Rake — all tasks which needed a lot of testing, coding and coming up with solutions.

For the Rouge switch, we've used the ALAN Docs repository as a test ground, since it offered more material for testing that everything worked fine. As it turned out, some docs (e.g. the B.Guide) couldn't stop using Highlight due to some custom formatting in code blocks which is still unsupported in Rouge.

For the switch to UTF-8 encoding, we've used the ALAN-i18n repository as our test ground, since it was smaller and easier to handle. As it turned out, we had to face various challenges that couldn't be handled via Bash/batch scripts, which ultimately accelerated the planned adoption of Rake for the toolchain.

And obviously, Rake had been in the plans for a long time, since it would solve so many of the current problems with using Bash scripts. But that required me to sit down and study the Ruby language from scratch, besides learning Rake.

As you can imagine, all these changes were significant, and required lots of tests, discussions, adjustments, research and study, which is why I decided to momentarily drop all work on the StdLib, until we had some working code prototypes and models to apply those changes here to.

Well, the time is finally ripe to bump up the StdLib repo to the new standards, so we'll finally be able to pickup development again.

The fact that all ALAN repositories are moving toward the adoption of commonly shared assets to handle the complexities of the various toolchains is an important strategy that will allow to reduce maintenance efforts and redundancy across these projects, making it much easier to setup new projects and update existing ones. Also, the new system is much easier to understand and use, since the more complex tasks have been abstracted away through interfaces to Ruby methods which require few basic parameters, and then handle all the magick in the background for us.

So I'm looking forward to carry out all the above transitions (it's going to take some time, for there are various steps involved), and then start working again on the StdLib code and documents. My goal is to be able to finish the long-awaited v2.2.0 release for this Summer, at least the code and most (if not all) the StdLib Manual — but then, it's hard to predict, since often time trying to fix a pending bug ends up revealing a whole new category of related problems, adding to the pending tasks.

Anyhow, for now let's focus on these major improvements listed above.

[AnssiR66]

Yes, sounds good. It would be great to have the update ready by next summer.

[tajmone]

StdLib Is Now in UTF-8!

@AnssiR66, @thoni56, I'm glad to announce that I've manage to migrate the ALAN StdLib repository from ISO-8859-1 to UTF-8!

The switch required some careful planning, because of the various problems that the adoption of UTF-8 encoding brought to the toolchain, so I decided to break the upcoming changes into individual steps.

E.g. I have to use Rake to make the test suite work in UTF-8, but right now Rake is used only for the test suite (I'm already noticing that running the entire test suite has become much faster).

It's a single commit with 169 changed files — locally there were multiple commits, one for each step, but I squashed them into one big commit since this was the only way to ensure that every commit will provide a fully function snapshot of the repository.

The next major step is switching from Highlight to Rouge, and this will require some careful planning too, as well ad replacing part of the Bash-scripts in the toolchain with Rake task. Again, probably another huge squashed commit involving hundreds of changed files.

Only then I'll be able to start migrating the whole toolchain from Bash script to Rake tasks (and even this will be a multi-step process, because getting rid of the SED sanitizer means I'll have to replace its job with Ruby code).

It's nice to see that all the major ALAN projects are starting to converge more and more on shared standards, conventions and assets for managements of their toolchain, documentation, etc. This is definitely an investment of time and energy that will pay off in the long term, reducing future maintenance costs for the ALAN ecosystem.

I also believe that it will have a positive effect as an incentive for new ALAN projects, especially from third parties, since a robust and well organized ecosystem always inspires participation.

[thoni56]

StdLib Is Now in UTF-8!

@AnssiR66, @thoni56, I'm glad to announce that I've manage to migrate the ALAN StdLib repository from ISO-8859-1 to UTF-8! ... It's nice to see that all the major ALAN projects are starting to converge more and more on shared standards, conventions and assets for managements of their toolchain, documentation, etc. This is definitely an investment of time and energy that will pay off in the long term, reducing future maintenance costs for the ALAN ecosystem.

And you are the Force behind it all! Thanks, @tajmone!

[AnssiR66]

Thank you Tristano! Great news 👍

[tajmone]

Done!

@AnssiR66, @thoni56, switching to Rake took me much longer than expected (about six days), since I had to adapt our custom Rake modules to the needs of the StdLib repository, but at the same time ensure the same modules will still work with the ALAN i18n project.

But it was a good chance to learn more Ruby and Rake by getting my hands dirty!

And it was well worth the efforts, since now building and updating the whole repository requires a simple rake command that can be typed in any terminal (CMD, Bash, etc.) form within any repository folder — no more Bash requirement, or external tools like SED, it's now all done in Ruby.

And of course the toolchain is now much faster and easy to use, since it launches less processes and only updates resources which need to. In this respect (i.e. tracking requirements), Rake is really a marvellous tool, for it makes it really easy to ensure that our documentation is, for e.g., updated whenever the external ALAN source examples it depends on, or their solution files (or the StdLib) are changed.

So, now typing just rake is an operation that guarantees that everything that needs to be updated is actually updated (no less, no more).

Of course, whenever we switch to a new ALAN release it's always safer to force rebuild the entire repository (via rake -B).

In any case, I've added the new instructions to the main README file.

I'm really glad we ultimately ended up using Rake for ALAN projects — the Bash scripts were starting to get a bit entangled, and hard to maintain in a way that would make them shareable with other projects (possible to do, but hard).

I had initially considered using CMake for the task (bought and read whole book on CMake), and although it was better than GNU Make, it wasn't too easy to adapt to our needs, whereas Rake turned out to be the ideal choice, since its just a Ruby base DSL, where you can use Ruby to handle all special needs.