Ideas and ways of development

Unfortunately, there is currently no serious site generator based on org mode. Most don't provide anything very different from the standard org-publish. But they carry all the same shortcomings that do not allow productive publishing of web pages that meet modern standards. I'm talking about html templates, schema.org markup and required meta tags. It looks like developers are anything but web. They didn't set the task that the published site (as a result) should be convenient and understandable for both visitors and search engines. Why then do we need these tricks with special static generators when your task is simply to publish a few of boring html pages? use your simplest project via org-publish.

Many site generators are characterized by the following shortcomings that hinder development (and, by the way, suppress the development of the generators themselves from the very beginning).

themeing. the need to customize themes, macros inside templates. An unnecessary layer of complication (hugo, zola, the vast majority).
A limited set of template languages or (oh my god) inventing your own. In addition to an over-engineered set of related templates, some manage to invent their own language for templates (hello weblorg).
support is not written in Lisp. => Most likely you will have to delve into the syntax of python, javascript, rust. Incredible brake in the ability to modify anything. Look for freelancers, programmers, if you want to change something, redo it. The most interesting org-tailored lazyblorg is written in python.
The generator solves basic tasks well, but the plugins/filters for the generator need to be written in a language different from the language in which the generator is written (for the soupault generator, you need to write filters in lua).
some generators dictate the structure of the future site, urls, based on their code. What folders and where will be located, where you need to put the content, in /blog/ in /content/.

The most interesting, from my point of view, generators that support org..

You can look at the ideas of a self-contained soupault (on ocalm) , which does not work with themes, but focuses on changing the html tree itself. You can modify and add to the original template by adding and changing html tags. Unique approach. Accepts all formats using an external converters, but you need to configure the conversion process from org-mode to html. And write filters in lua for pandoc.
lazyblog tuned for org-mode. It parses posts from .org files with a subtree tagged with a specific tag. Uses simple templates. But is written in python. There are restrictions on the site structure and templates. Doesn't work under windows.

To make a .org site up to date, now you have to fiddle with a well-known (and usually bloated) site generator, which will:

ensure compliance with standards. Out of the box or with plugins.
be able to get files to the converter/plugin to send them to the generator. I conclude that then it makes sense to dive into a generator written in javascript and jump to jamstack bandwagon. At least the need to study one more language will expand the possibilities of application. Since js is the language of the browser anyway.

If you create content in org, emacs and want to make a working site that receives visitors and performs your goals, then now you have to deal with the following paranoia: fine-tuning org (elisp)-> the language in which the generator is written + perhaps a converter (pandoc, encoda) from .org to introductory markup for generator (usually markdown) -> learning the language and rules when modifying themes and templates (pug, mustache, etc.)-> modifying html->modifying and optimizing css.

Adding to this we can add the need to connect javascript inserts for interactive elements (forms, surveys) and the need automate these modifications by page type.. Is it possible to complicate the process even more, otherwise it’s not fun enough?

Right now I have this wild publishing sequence:

python the script looks for subtrees in org. files marked with a special tag. Folds them into separate .org files.
pandoc converter with lua filter converts html files.
soupault generator connects everything to the site. Schema.org microdata issue not resolved. Flaws outweighed the benefits. It's not worth continuing.

Another 'simpler' scenario is being considered: python script posts from subtrees org file -> astro generator (written in js) with pandoc plugin converts org and generates html. At first glance it has the necessary plugins.

A serious generator for org should be in Lisp and integrated into emacs. It should not dictate the structure of the future site, nor the introductory files as 1 file per input to 1 page of output. Have the ability to publish posts from subtrees from within files has many advantages.

When the article becomes a tightly connected part of the informational project management process in emacs, and not as a separate file that needs to be specially prepared. Get away from levels of sophistication such as your own templating language and themes.

At the first stage, should not engage in expanding of generator's features , but to achieve such a result of the whole process as the publication a real working site that meets the standards in one method. Then expand. See the ideas embodied in the most interesting generators.

The most important thing that potentially gives emacs and org is integrity workflows => increased productivity. Study and immersion in development of such generator is a deeper development of emacs at the same time. What is completely worth it. The publication of posts becomes closely associated with actions preceding their appearance, todo organization and info management into subtrees.

Real problems solved by individual plugins and add-ons in other generators are partially already resolved in emacs. At the same time, emacs is much larger and is used for a much wider range of tasks than any of these generators. I don't know not one of the generators or cms, which would have a practically operating system (emacs) somehow in the 'back-end' .

If your project sets itself such goals, then it will be something special. I will be the first who is ready to support such a project. What it takes to try cl-ownpress and will it work under windows?

Thanks for sharing your ideas in such detail @MorphicResonance I can relate how you feel when it comes to complexity added by static-site generators and the benefits we get from them.

One of my objectives with cl-ownpress is to actually utilize Emacs/org-mode to create my website(s), not just use org as a markup. So far I've been successfully using it to publish my blog.

cl-ownpress uses my running Emacs instance to convert .org file(s) to .html; it basically does org-html-export-as-html but on per-file bases (instead of project). My problem with org-publish-project was it being super slow and very hard to extend. I did an experiment with it though, you can check it in press.el repo.

Although fully utilizing Emacs is one of my goals, reducing the complexity is not (for now). I don't mind complexity at this stage as long as it can get me what I want; which is hackability. But for the first "theme" I am building in cl-ownpress, everything is written in Common Lisp (i.e css, javascript, html); which might actually seem more complex to some eyes.

How to try cl-ownpress

cl-ownpress is written in Common Lisp, which although a lisp is not Emacs Lisp. So you need development setup for CL (Common Lisp) set up to try cl-ownpress.

Install sbcl
Install quicklisp
Install sly (or slime; I use sly) in Emacs
Install sqlite.
In $HOME/quicklisp/local-projects, do git clone https://github.com/bitspook/cl-ownpress
Anywhere else in your computer, do git clone https://github.com/bitspook/bitspook.github.io. This is my personal blog you can use as a starting point to play around with cl-ownpress
Open site.lisp in Emacs. And do M-x sly (or slime); this will start a common-lisp repl in Emacs
You can now either do M-x sly-eval-buffer, or go to individual lisp forms in site.lisp and do C-c C-c (or C-c C-e) on them to run each of them at a time. You might want to delete form at L:17 which publishes my denote notes.
Last form in site.lisp (L:32) builds the blog in ./docs directory. You'll need to start a static server in docs directory. I usually do python -m http.server; you can then browse the site on whichever port your static server is running on.

A lot of steps if you don't have a CL dev setup already :smile:

If you end up trying cl-ownpress out and have any problems/questions, please feel free to ping me. You can create a comment on github, or dm me on matrix.org (e.g via app.element.io) or twitter even. My handle is @bitspook on all three platforms.

PS I don't know if it works on windows, but I have no reason to think that it won't either.

Researched the github for what software packages connected to emacs org-mode are written on. At the first place is for elisp (not surprisingly), further clojure, python. Common lisp before occupies a higher position, now software is found with packages written in 2010-2020 Software for org-mode that are written in Clojure is growing.

Required combination of software modules built into the workflow look like below. endeavor adding things as examples about which you wrote on your website or their alternatives. By the way, why not add a contact form on the site.

content mining and collecting. Collect pieces of content from different sources for deep processing into org-mode or publishing as is.
- Automatic/semiautomatic. There are very few examples in this field. The most ambitious here is this orger (python based). Collects information from many sources, accumulates in org.
- Orger: plaintext reflection of your digitalself, beepb00p.xyz/orger.html
You can add automatic mining of the necessary data during preparation periodically updated content. Example: 'Consistent Technical Documents Using Emacs and Org Mode', reddit.com/r/emacs/comments/dxk4py/consistent_technical_documents_using_emacs_and/ Add in this category api, http requests also.
- Mostly manual. Here you can also add org-capture, manual notes from pdf, video. Some examples:
- org-media-note: Taking interactive notes when watching videos or listening to audio, github.com/yuchen-lea/org-media-note
- collect org bookmarks from browser, github.com/include-yy/yyorg-bookmark
generating content in emacs Here is the widest field, in fact, this is the main focus into emacs org-mode. From codes in org-babel, to creating presentations in revealjs, graphics, pictures + text and so on. Example:
- generate tweets from emacs, github.com/neil-smithline-elisp/org-status
converting and publishing content to html and numerous others. Let's stop here only on converting to web pages. Actually org-pubslish itself with some additional elisp functions could be sufficient in some cases. When not enough, let consider static site generators preferably in lisp languages. Discarding the opiniated static site generators that are abandoned and do not support org-mode. Seen 2 types.
- add-ons for org-publish:
- Pile Personal website generator based on Org mode, github.com/lepisma/pile
- minimal org-publish in common lisp, github.com/ahungry/ahungry-blog

not based on org-publish , but matching the criteria of 'not creating problems'.
- just pandoc converter itself with lua filters/plugins. thinks It's probably faster than org publish in terms of speed, but delving into filter settings in lua language is not a desirable path.
- Shite (bash script+pandoc for conversion), github.com/adityaathalye/shite. Unknown if it can work on windows. However, a well-thought-out approach many problems rather than building them up.

It is not surprising that the greatest interest and coverage of the audience will have software, written in the eLisp language to interact with org-mode.

Moreover. Page generation speed can be a problem for org publishing, but it's not necessary to generate an entire site when changed just a few files. It is necessary to detect which files have been changed since the previous generation and generate only them (solved lazyblorg ). It's easier to use a serialized array as a database. I used this one with a small php cms site. Worked fast with over a thousand pages.

Hey, Just wanted to drop a message that I haven't yet looked closely at your above comment yet. Right now I am held up by life and will take a closer look and respond once I can get a moment of peace.

Summing up and clarifying what has already been said above, it must be said that creating yet another static site generator with Lisp is an error. Creating of just static pages is good suitable for ox-publish.

However, in order to make a visited site that meets many requirements of the modern web, you will have hard to work with org-publish. Implementig of many lisp hacks and javascript for forms + not native system of commenting etc, rss, indexes. These efforts are getting closer to learning another generator. But it is not a solution. It’s also not will give dynamic functions for your site => you 'll be rolled down to a pile of different languages , heterogeneous codes supported by different developers. All that you need to follow, update. Yet Another generator static sites, even on lisp? There are many similar. Conclusion: for a simple static site like publishing of pdf ox-publish is enough (included in emacs and will be supported/updated guaranteed).

For more advanced site it's needle of static-dymamic generator, parts of which will run on the host. With it commenting, sending a message via contact form and more can be done self-hosted. And on the same lisp.

But to make a generator with dynamic modules that would connected to static pages is another matter. We can generate static pages and connect the necessary dynamic parts to some of them. Let’s say it can somehow be combined with your brandI. So we got the advantages of static pages + solve issues with the necessary functionality for defined static pages. Need consider each page as a separate functional unit with assigned property.

Only such a performance is unique and it is a step forward.

And in this regard it becomes clear why you need to write not on elisp, but on common lisp. To make it work well on the server, there are many libraries, not confined to emacs. At the same time, these languages are in many ways similar, only common lisp is wider. And writing html code on lisp is also justified, if we talk about uniformity and productivity. Need to try work with the same objects everywhere. Although this can complicate understanding.

With experience in the field of promotion of niche sites, I note that from the functions want to see in cl-ownpress, they are (in order of importance):

shortest memorable URLs are important such as https://site.com/my-page
schema.org markup support. is necessary for positioning in search engine page results. Don't want publish the site without it. Re-indexing can take a long time, months.
themeless. HTML code modification instead of complicated theme with partials/nested templates Modify each page by destination during generation. You can add tags/replce, etc., by modifying of the base template. Implemented by y soupault generator. Connecting dynamic modules to each page. We also assign on which pages we want include comment forms, contact form etc. Thus, when page generation is generated it goes through a series of filters and gets its new properties.
receiving content/comments from a certain external web page. That is you can start a post somewhere say on the reddit and take the active discussion for your site, update it and add more content, pick up user's comments. It's possible to do with YouTube and others to. And instead of seeing how your posts is getting outdated in listings of social sites we get proven interesting buzzing topic on own website and make it maybe evergreen updated. It can be done with days instead you can wait for user reaction through comments on the post of your site for months.
commenting system on website. Famous thing. It would be good to do support for anonymous users can comment but ignore spamers.

This kind of static-dynamic generator is the most expected thing for self-employed content publisher.

Thank you very much for sharing your thoughts @MorphicResonance! Thank you for adding links to the various resources! I didn't know about projects like orger and memacs. They look very good!

I do intend to build something similar. You are also spot-on in relating brandi with cl-ownpress. There are 3 systems I am building in parallel:

Brandi: for persona management. A private tool which allow a user to manage their online data. It's most similar to HPI in that regards.
Saunf: for project management. I use it to make my interaction with my work (software development) smooth.
cl-ownpress: for publishing. I intend to use it to publish a subset of my personal life as a public website (at least).

I also like the idea of a static-site generator which augments the static pages with some dynamic functionality. I experimented with that in Entropy; and I do intend to evolve cl-ownpress into a better version of that.

Current version of cl-ownpress had one goal: create bitspook.in I have cut a lot of corners for that which I am fixing slowly.

For the stable version of cl-ownpress, I imagine 3 abstractions:

Providers: An object which "provides" some data. It might get it from anywhere, save it anywhere, does not matter to the system.
Publishers: An object which creates publish-able artifacts (e.g HTML page(s)). It accepts other publishers (e.g for linking to content produced by them) and providers.
Widgets: An object representing a fragment of a publishable artifact. For web pages, it is an object which can produce HTML, CSS and JS for a part of an HTML page. You can think of them as React.js components (or web components).

And I want all three of them to be compose-able. Imagine you create 3 providers, for markdown files from a directory, org files from your notes, and for toots you published on mastodon. You should be able to combine them into a single provider which you give to your home-page-publisher. home-page-publisher uses this provider to create a "Recent content" section on the home page, and pass it along to all-posts-publisher. all-posts-publisher publishes many HTML pages for paginated lists of all your posts, and provide a link which home-page-publisher can then use for creating a "View all posts" link.

soupault's approach of modifying generated HTML via CSS selectors look very useful. But I have a slightly different approach which I am working towards for now. I want widgets to be "hack-able". A publisher uses widgets to produce publish-able artifacts, and as a user I should be able to change the HTML, CSS or JS produced by a widget. I think CL's generic functions provide this ability for free, we just need to come up with an abstraction which can utilize them properly. For example, imagine you added a new section to your personal site. You created a new publisher to produce super cool adventures, but you also want to highlight them on your home-page. So you create a new widget for "Latest adventures" section, and tell recent-posts widget of home-page-publisher to render latest-adventures before it.

That said, I do not concern myself with the matters of practicality. Other similar tools exist, I will try to use them where it makes sense (most probably HPI and dogsheep, thanks again for bringing them to my attentoion!), but it does not affect my work on cl-ownpress in any way. I am building this because doing so brings me joy.

Just recently spent many hours researching the topic of choosing a programming language that must fit well with Emacs, support simple web development projects (as static pages and dynamic server pages), have future perspective ideologically revolves around simplicity and implementation for single user. This short list of applicants includes three languages: emacs lisp, common lisp, guile scheme.

And it turned out the following. Emacs lisp is somekind of 'local' internal language of emacs. Yes it is similar to common lisp, but it is incomplete. In principle elisp does not support many things which are common to common programming languages. Using its outside of Emacs, you can face unsolvable problems. For web development it is very limited, otherwise it would be already used, only as a static site publsiher. Many say about the obsolescence of elisp language for emacs. Its future uncertain. However, no one is going to rewrite tons of packages' code, and so another language will use the connection to elisp and its control.

Common Lisp seems the most logical choice, but there are obvious votes against. Yes, it has everything you need and even can imagine. It is bloated and it is not in its favor. CL is industrial language with a long history. its learning is overkill for highly clear tasks of computing and web pages publishing. Several static site generators are written with it and there are some examples of web development exist on github (started many decades ago). CL connects well with Emacs. Other powerful open source software are written in common lisp, but emacs uses limited elisp as kind of lisp family, unfortunately. As far as I can see, GNU is not going to rewrite emacs for common lisp, or create new projects in common lisp. Otherwise they would have already been done.

Yes, there is development of the emacs version in common lisp such as Cedar, but it’s not from GNU, it’s an activists' initiative. So the future of gnu emacs and other powerful stuff for developers from them is not in common lisp. They defined the language in which they release programs as Guile Scheme.

So studying common lisp for the above purposes maybe is not the best idea. Yes, there are cases when good software appear on common lisp such as Nyxt browser. And user can get an added benefit of knowing such common lisp coding. Hower I would prefer have a functional browser in emacs (eh, someday it will happen).

Guile Scheme is the most lispy language from all schemes. It seems to meet a lot of the requirements. First, elisp can be used via guile outside of emacs. Since Guile Scheme is the GNU language in which GNU software is written, 'linking' it to Emacs will grow. Generally it is simplier, works for personal computing, which is required. It has web development tools]. Found several static site generators were written with guile. Also, by studying it and using it, user expand his capabilities to explore other software from Gnu (Guix operating system). For the stated goals and prospects. It is worth mentioning the comparison of common lisp and scheme in the expirienced developers' article.

Moving on static sites generators with an attempt to discover the expansion prospects for dynamic pages. All generators are written in common lisp or guile scheme. What characteristics have been taken into account: readiness for work and availability of example sites, export/support org. files, documentation, expandability for dynamic functionality.

|_________________________________________________________________________
| name                    | ready  | docs        | language written | org-mode support    | dynamic extension | notes                                    |
|_________________________________________________----------------------------------------------------
| site-generator          | yes    | yes,        | common lisp      | yes, through pandoc | no        | explained lisp in templates                      |
| sph-web-publish         | yes    | yes, short  | guile scheme     | no, markdown        | yes       | webapp for dynamic and sph-publish for stat      |       
| hackable ssg            | no     | no, notes   | common lisp      | no, possible        | yes       | super extensible generator. site as lisp library.| 
| haunt                   | yes    | yes         | guile scheme     | yes, ox-haunt       | yes       | has examples.                                    |   
| Parenthetical Blognir   | no     | no.blogpost | guile scheme     | no                  | yes,fully | dynamic blog. underdeveloped                     |
|_________________________________________________--------------------------------------------------------

Althrough, I've found an explanation of writing templates with s-expressins in common lisp. People talk about splitting the variables.. See manual for AlexCharlton's site-generator.

Looks like haunt is eligible for org-mode/emacs requirements. Sph-web-publish with sph-web-app are interesting as an examples of ready-made system implementation with static and dynamic parts. No sub-detailed description complicates its learning.

Haunt and Hackable static site generator are positioned as very extensible. Haunt has an exporter for .org files called ox-haunt. The first dynamic component - the comment system experimentally was already written for it.

Hackable static site generator is most interesting because potentially it meets all the requirements. The author poses difficult problems when writing it and have details about his way that can be useful. It is written in common lisp. It does not yet work with .org files, but he’s told it can work with anything.

The out of table generator is Firn. It focuses on . org files completely, takes structure and links. Unfortunately, cannot be explored under windows. It’s written in rust.

Originally I inclined to study guile scheme because it will be used as the main GNU language. However, since common lisp has a lot for web programming, it is more distributed and other necessary programs such as nyxt browser are cl based, this outweighs the 'loading' of the language. With guile there is a risk of not find solutions to questions expecially regarding dynamic parts of the site. So Common Lisp is my choice.

Data transformation.

Here, when developing, the basic task is to reduce the number of intermediaries in processes. Something complete and well managed by one human only can be made by simplifying the chain of actions.

How generator will transform the data and in what sequence is important. It would be good to get data from .org file and translate data into json machine highly readable format and then translate it into html. The encoda converter is working so way.

In discussing the experience of creating a static site generators, there is a comment that you need to parse data first in json format then convert it . How do geneartor take the data and in what sequence it transmute, how the data will be fragmented, then how to get new and custom structures into site design , where is the key problem. it is necessary to transmit it to metadata and this case can be more complicated. That is, ideally it should remain path to connect other input file formats into the generator, but each such format can be too much with their own characteristics. Such focusing on this service takes away from the purpose of creating an effective workflow. In this case pandoc can play hard.

How to mark content in org-mode to get the html output according to and compatible with schema.org?
Typically, the finished page should be edited again by filling in the fields of the form (rankranger.com/schema-markup-generator) and you'll see the outpout as script/microformats. It should be embedded in the page. Doing this with each page.. , it seems unproductive spending of the time. With other formats as markdown we have another fun, offered by writing the .yaml headers with nested types, properties (hello to sambal ssg).I’ve seen some partials for hugo, 11ty for that

In org-mode, this can be simplified by using its default tree structure. For example, the usual schema.org type for a blog post is blogposting . Create a site structure with inheritance of properties. Write properties in org header, they are inherited. Somehow. The properties should be nested. There is still maybe incontent inline markup required.

* my blog section of the site
:PROPERTIES:
:CREATED:  [2023-03-31 Fri 01:09]
:schema_type: blogposting
:End:

** my post title
:PROPERTIES:
:CREATED:  [2023-03-31 Fri 01:11]
:headline: my post title
:datePublished: 2023-03-31
:dateModified: 2023-03-31
:End:

This is shema markup layer of the site. You can write own properties depending of the context. To these properties you can add a layer of properties for each post or part of the post. For example, some characteristics of the object described in the post. And these characteristics can be written from different angles also. That’s where the power is.

The source language is correct, but exporting from org subtrees and avoiding the bloat database sql format is what I’m waiting for from your generator.

Really interesting reflexion here, thanks @bitspook @MorphicResonance /o\ and i just found your project @bitspook , this is great because it share some of my interest, i need to explore it !

I'm definitively like you @MorphicResonance, i search the good tool for my workflow. I try to describe it in multiple way (here https://org-roam.discourse.group/t/advice-on-roam-workflow-how-do-you-separate-private-and-public-part-of-your-garden/452), on org-roam (https://github.com/org-roam/org-roam/issues/911), more recently on an issue on astrojs and uniorg (https://github.com/rasendubi/uniorg/issues/82).

In two words :

common lisp
all features of org with literate programming : org-babel, noweb, org-transclusion, dynamic block (https://github.com/org-roam/org-roam/issues/2251)
one database of notes (denote/org-roam/whatever), but fine-grain routing, with multiple output possible.

I found that "identities" is a really really important thing in our modern society. This is the way for better privacy. I want to manage/hack/say about my different life the way like i want. One individual, one database, multiple publishing output.

Astro JS + Uniorg really good candidate SSG for org if you accept to publish using js (https://github.com/rasendubi/uniorg/). But i prefer common lisp, like you, and there are no such equivalent tool (perhaps cl ownpress!) actually.

There is also Vulpea in ELISP : https://github.com/d12frosted/vulpea

( we need to put all our sources/tools on a website @MorphicResonance !)

Ps : https://infosec.exchange/@bitspook/109353596205512997 on the readme don't work :)

Hello @reyman !

Glad you like the concept. From the links to discussions you posted, it seems like you want pretty much the same thing as me. Publishing subsets of my notes to different places while making full use of all the customization I do in my computer/emacs.

Updates for the current state of cl-ownpress:

My current focus is to implement a cleaner base API and have good tests for it. I am currently focusing on the html publishing side. Progress is slow because I have this bad habit of taking my work 'personally'; so that means little time left for personal projects.

Ps : https://infosec.exchange/@bitspook/109353596205512997 on the readme don't work :)

Looks like I have auto-delete on for old posts 😑 Thanks for letting me know! Maybe this will give me the push to finally implement 'publish micro-posts to my blog first and mirror to mastodon' 😢

@MorphicResonance Thank you very much for sharing your thoughts, and also for linking some very interesting projects! Your recent comment got lost in the flood of notifications; I apologize for not getting back to you sooner.

From the cursory look, cl-hssg seems to be pretty much exactly what I want with cl-ownpress. I will see if I can get "inspiration" from it 😉

Idea of generating an agnostic middle format (e.g an AST in case of pandoc, or a json schema) is appealing; perhaps that's what I'll get to in future, but right now it seem distant.

I hope I can get cl-ownpress to a point which is useable for you; that'll make me happy.

Thanks again!

Looked at your blog and once again convinced that we have similar goals about crafting tool for creating and publication of textual content. It is a pity that the movement is slow , a year passed. However what is already done .. must say that serious beast is born.

It's good that the initial thoughts led to quite defined and described concept Extract Transform Load .

And I want to define a whole workflow as I see it. it consists of 3 parts: collect/write (Extract), edit/generate (Transform), publish (Load).

1) Extract. collect/write search and collect info on your pc or/and online. Write something. This also includes the collection as you wrote in the past from different personal disparate sources such as online messengers, tweets, emails, youtube channels, feeds.

About searching for information on a personal computer and online. There are Emacs packages for it also. Small breakthrough appeared as Consult-omni package. It allows you to look for information at once in many sources, dynamically examine the desired sites. As I still stuck on Windows therefore failed to tune it fully. And the EWW browser loads pages for a long time. But in general, a very sensible package.

About collecting of info from websites. There is your spookfox. if manually, through the key control like Surfingkeys or Nyxt browser with its keyboard hints, you can highlight the text on the page and copy. Here, of course, there is something to prepare to get the right paragraph directly to the Org file.

2) Transform. Generate and edit. Give shape to the information. Paragraphs, tables, graphs, pictures, additional material. This is where Emacs and other so called "providers" come to scene. See you have plans for connecting HPI as provider. From the source of your site I see org-file-provider, org-project-provider. And cl-ownpress itself have emacs-provider.

Does this mean that org-pervider sets the design method, and Emacs-provider is responsible for processing. Is that you can write an org-one-pervider and publish posts from 1 input file?

And it seems that CL-ownpress will not be limited to publication only in HTML, but it will be possible to set a transformation method from input data and get the plain text format also and the other Types than the initial data set. I also have developments in this part and I realized that my script can exist as a provider too. Contact me through contact link in the profile for details. Unable to reach protonmail..

3) Load. Publish and distribute. A complex part. It is called to attract new users to content. As a result, users with similar interests will come to this content. Decide what to do with them next.

If the publication does not occur on your own site as html pages, then output ready for publication in social networks/platforms/forums should be just textual, or formatted as social networks support it (some stick with markdown). The publication can be semi-automated through Emacs controlling browser. With plugins such as chrome-emacs, atomicchrome, ghosttext, emacs-with-nyxt.

Regular publication of content is important part of development of digital asset.

Regarding the markup, we need to deal with templates and metadata. How to get it and where. For example, the text in the code is immediately included in your About-Me page. In theory The text should be taken in the same way for all pages. Such as also from the About-me.org file.

What you turned with widgets by creating different kind of listings. Of course, it leaves the org-publish to the side.

One Part of the metadata is static, other such as the time of publication and other contextual things changes from the page to the page.

By the way, a correctly marked and structured website is its own passive promotion.

By marking using schema.org. Introducing of page types according to Schema.org is not a problem. These may be in widgets. But to mark the context inside each post with itemprop tags. This can be done either with text processing API, or manually by assigning org heading properties. Even if we find and connect 3rd party text processing api, it will find only those things that exist in public dictionaries. And not for domain specific entities. It remains us to set the entities in the published texts through org properties manually.

Here is a dirty scratch how it can work as an example of page about the movie. How it can be marked.

#+TITLE: 5 cool movies about pirates
#+PROPERTY: schemaorg_type BlogPosting 

Here are some movies about pirates worth watching.

* Pirates of the Carribean: On Stranger Tides (2011)
       :PROPERTIES:
       :schemaorg_@type: Movie
       :schemaorg_name: "Pirates of the Carribean"
       :schemaorg_director: (:@type Person :name "Rob Marshall")
       :schemaorg_description: "Jack Sparrow and Barbossa embark on a quest to find the elusive fountain of youth, only to discover that Blackbeard and his daughter are after it too."
       :schemaorg_author: (:@type Person :name "Terry Rossio")
       :END:

Jack Sparrow and Barbossa embark on a quest to find the elusive fountain of youth, only to discover that Blackbeard and his daughter are after it too.

 Director: Rob Marshall
 Writers: Ted Elliott, Terry Rossio, and 7 more credits
 Stars: Johnny Depp, Penelope Cruz, Ian McShane
  8/10 stars from 200 users. Reviews: 50.

If bring types and itemprops in the :Properties: of Org-Heading as nested properties. Then extract them and replace conformities through the replacement table similar to this. Replace the one once sought out for markings.

As a result, marking should be obtained as in the first example #1 at the end of the page. Take a look.

Org export allows you to set your tag in which org-heading will be wrapped as html by using properties.

    :HTML_CONTAINER: div itemscope itemtype="https://schema.org/Movie"
    :HTML_CONTAINER_CLASS: TEST

But Org-Export by default produces some kind of garbage with this.

When we prescribe data to the things we need in the text in their org-headings, we get a dataset or even database, dissolved in our texts. This can be used many times too.

By the way, for the names of widgets or providers, I recommend turn head to the defined terms of schema.org classification. There is completely certain classifications for post, article, and code also.

I'll try to study in more detail how it works with an example as your src site. Do we still need to install SQL to make it work?

Regarding that such information should be on own website. I keep posts locally before publishing online.

Glad to progress.

Hello @MorphicResonance!

Thanks for your insights. I've put together an example to document using cl-ownpress. Hopefully it will answer some of your questions. I hope you can go through it and help me improve it to make it more useful for you.

I am building cl-ownpress as more of a library which can help build 'Publish workflow'-s. We can definitely have something like 'Publish a schema.org compatible website' as a separate project which uses cl-ownpress, and add utilities/abstractions in cl-ownpress to support that.

bitspook / cl-ownpress

Ideas and ways of development #1

How to try cl-ownpress

Data transformation.