Exported with 1.8.1-2b HTML file size became extra large compared to previous version 1.7.3

[vgyenge6]

Is your feature request related to a problem? Please describe.

I do not know exactly whether it is a bug/feature request or from what this problem is arising, but I noticed:

When I exported the same note file(s) (I'm speaking about exporting an individual file or files with HTML export preset) with the newer 1.8.1-2b Pre-release version the file size became extra large. For example, the same file that I exported with version 1.7.3 its size was formerly 232 kb, but became 2.3 M exporting with 1.8.1-2b. In the meantime, I only added some relatively short script.html with this new pre-release version to the Custom head content (see: https://github.com/KosmosisDire/obsidian-webpage-export/issues/387#issuecomment-2022227600) which did not affect the exported file size having no before this addition. I also tried to set it to Minify HTML, but that also did not affect the file size.

Note: I installed previously the intermediary 1.8.0 stable, but I ran into a problem due to the obsidian 1.5.8 change, thus I had not observed then whether the file size became larger exporting with that version because I immediately deleted them, and I reinstalled the former obsidian 1.5.3 and your plugin 1.7.3 version. Then came the 1.8.1-2b with special fixes for relative TOC links that I needed (see: https://github.com/KosmosisDire/obsidian-webpage-export/issues/366#issuecomment-1972943205) and works well in this pre-release version, thus I have been using this version since then running under obsidian 1.5.3.

Thank you for looking after.

[KosmosisDire]

Hi, this is probably because I am now packaging fonts as well. I'm trying to get it smaller, but I want to prioritize it actually working haha. Most of the fonts aren't used, so the goal would be to detect which exact fonts actually need to be packaged (or allow the user to choose) which would make the file size smaller again.

[vgyenge6]

Hi, Thank you for your answer.

I can't estimate the resolution, but, please somehow resolve this as soon as you can, because if I upload those HTML files to (for example as I mentioned I wished to use them on a ) WordPress host its storage place will run out quickly, and the big files slow down the site performance. I think this may be a problem not only for WP but also for else web hosts.

Anyway, I'm waiting for the resolution. Thank you.

[KosmosisDire]

These files are meant to be opened locally, why can you not use the online web server export? How are you including these onto your site?

[vgyenge6]

hi,

why can you not use the online web server export?

Because I think, your webserver export can not upload to a WP host site due to several security constraints (see: Publishing directly from Obsidian to WordPress – James Whayman. My site is not hosted on the standard WordPress.com, this latter even needs an Obsidian plugin. Thus, I manually upload my locally exported html files to my WordPress host media folder and I'm displaying them in an iframe.

[vgyenge6]

Furthermore, I think, although your plugin's web export feature mainly follows the structure and logic as Obsidian's publishing however your plugin's great advantage is to export individual HTML files assembled arbitrarily (I mean having both left and TOC navigation if I want), or I can export individual HTML files and be uploaded individually and structured by an arbitrary navigation by the site logic. (I want to use WP because it has tagging categorizing files and I want to navigate by my site logic those posts or pages that embed in an iframe my otherwise written and gathered topics in obsidian and exported those to HTML, WP also has in-built commenting and many useful features.)

[KosmosisDire]

Have you actually tried? The web export also exports html files that you can simply upload to the WordPress server and then open using an iframe.

[KosmosisDire]

The only difference is that the web server exports contain a bunch of optimisations that rely on a web server to host them. As long as they are on a web server it's fine. I am not 100% familiar with how wp works, but if you can upload self contained documents you can upload the web server documents.

[vgyenge6]

Have you actually tried? The web export also exports html files

Not yet. I was reading earlier 5. Export Modes and I misunderstood (?) the web export will publish/upload/save the files to the web server if the Public site URL has been given in the Settings Metadata section.

If the web export will publish immediately to a web server, that seems sure I can not give the WP site URL and can not online publish/upload/save with your plugin there. To reach the WP site requires special authentication and there are more restrictions as I have given the (see: Publishing directly from Obsidian to WordPress – James Whayman link. (I did not try that plugin either.). Furthermore, WP can not hold that directory structure as how it will be otherwise published normally to a web server. Thus, it seemed to me simpler to export the HTML files locally and then upload those files to the WP media folder.

Does your web export save the files locally to my computer or to where exactly?

What do self-contained documents mean exactly?

How do you mean exactly >bunch of optimisations that rely on a web server ? What kind of optimizations those are?

Why can not be locally saved files optimized too?

Thank you for your kind answers

[vgyenge6]

again, your plugin's great advantage is as I see that if one uses the HTML export to save locally it does not mean necessarily he does not want to host those files on a web server, or else site e.g. WP later. Rather, this possibility allows arbitrary directory structure and navigating documents that had been uploaded to a web server, not necessarily following the logic of Obsidian's publishing site navigation and vault structure.

[KosmosisDire]

The web export literally just saves html files onto your computer just like the HTML Documents mode. However, in HTML documents mode, css, js, fonts, etc will all be embedded inside every single file (this means a copy of all these files is included for every file, which is why the file size is so large). With the web server export mode the html files only include the html for the document, and instead reference the css, js, and fonts externally as well as things like the file tree (which appears on all pages but is identical). These things cannot be loaded on a locally opened files because of browser's CORS security restrictions (the file would have to be able to read arbitrary files of the users computer which is not safe). So the files must be served over an http server to be viewed.

The plugin does not do any kind of automatic upload of files to anywhere, it just spits out html, css, and js, in different formats that are optimized for use in different scenarios.

Incremental export just means that if you export multiple files to a location, and then you change one file and re-export, it will only have to regenerate the one you changed and not all of them.

[KosmosisDire]

I will try to work on clarifying this better in the documentation, it's still a WIP

[KosmosisDire]

Also, if you export all your files into a single obsidian export directory structure, you don't have to navigate it in that way on your website if you just reference the urls that you want on whichever page of your wp site.

[vgyenge6]

Hi,

https://github.com/KosmosisDire/obsidian-webpage-export/issues/415#issuecomment-2024106391

Thank you for taking the time to answer and clarifying to me the essence. It is clear, I understand.

I will try to work on clarifying this better in the documentation..

It is welcome because what you detailed above is very essential to clearly understand. Some screenshots for the different scenarios also would be welcome. Understood it is in WIP state.

[vgyenge6]

if you export all your files into a single obsidian export directory structure, you don't have to navigate it in that way on your website if you just reference the urls that you want on whichever page of your wp site.

Am I understanding well this means I must select the 'select all' which means my all md files are in the vault and if I choose the web export method the html, css, js, and fonts contained in them will be exported separately to a given/chosen directory in bulk?

If this is so, unfortunately, it is not a good strategy for me. Because I'm using a complex vault structure, (not separated into several vaults by means) and not every md file in the vault is intended to be published on a web server later.

However, I could see if I have chosen some selected files intended to export only from several directories being in the same vault the directory structure from which the files were selected, and also the navigation will be created even if I'm choosing an arbitrary export directory e.g. ExportedFiles. (until I have used only the HTML export method.)

Is it possible to export only some selected files chosen from several directories of the same vault in bulk (without retaining the directory and the navigation) into a given chosen export directory?

[vgyenge6]

going back to the original issue problem

https://github.com/KosmosisDire/obsidian-webpage-export/issues/415#issuecomment-2023080022

Anyway, it would be good to make the size of the self-contained HTML files smaller.

I do not exactly understand why packaging the fonts into the self-contained HTML is so essential, but I guess because one might use themes other than the default, and/or individual font style tweaks. (I'm using the Obsidian default theme and not any font style tweaks). Yes, perhaps to choose by the user which fonts to be embedded may be a good idea. But until it is fully developed and tested how the size can be smallest in my opinion, it would be a better idea to allow only to generate the self-contained HTML using the default fonts as a general. The smaller file size stored on local computers is also important. The extended file size because of this 10x, that is really large.

[KosmosisDire]

No you don't have to select all, as you said you can just select any files you want. Why do you need to flatten the export structure though? You can upload the structure as it is can you not?

[KosmosisDire]

Also please remember you are using a beta version. The font packaging is a beta feature and is not complete. Thanks

[vgyenge6]

Hi,

First of all, I very much thank you for dealing with my problem, and you are keen to respond. I'm not sure whether it is the right place to discuss these things in detail. I can see there is a discussion page you are running and I think perhaps that would be better to discuss such general or special use case issues there.

Anyway, I am following up to give my answers to your question here, and I want to be so simple I do not burden you with too many details because I do not want to distract you from dealing with development what regard to this issue only. But to understand clearly my concern I need to be a little longer and more specific.

You can upload the structure as it is can you not? Why do you need to flatten the export structure

My special use case is:

1. I'm building a personal PKM in Obsidian locally. It has several directories under a simple vault that covers many topics. I'm gathering materials and I'm writing notes and also such notes that summarize those topics. Those directories also contain else files as assets e.g.pdfs, media, etc. regard to topics. Even finally I would like to write books, perhaps not only one book but several books.

2. The more painful thing with Obsidian it does not give the possibility to export specific individual notes in HTML format, which can retain my styles, and I can not insert anyhow my scripts either into those exported HTML files. It is only your plugin with which I can do this.

3. Although Obsidian's publishing logic and structure regarding assembling an HTML-based website simply and with transparency is good and truly well thought, however, it restricts how a website navigation and logic can work.

4. I definitely do not want to follow Obsidian's publishing logic and navigation, but I would like to build a WorpPress website, however, I would like to utilize the capability your plugin can do with the navigation both left and right (TOC). You may better understand if you look at the picture of how my WP site will look. (This site is heavily under construction, so much is not finalized yet.)

5. You wrote you are not familiar with WP, thus I will go into the details as much as possible.

Regarding upload to a WP site: A WP site can not maintain an arbitrary directory structure manually created as that can be done on a web server and files uploaded there by FTP or else. WP has a special directory structure. The posts and pages are basically HTML files but are stored in a WP database. One can create posts and pages using the WP UI posts and page creating, and editing capabilities only. Because WP pages and posts are HTML files one can embed externally created HTML files either using an HTML container or a simple iframe container in a post or page. I'm using a simple iframe container. In this latter case to display this externally created HTML file(s) I must upload it/them to a special WP folder called the media folder, technically it is a built-in WP directory/wp-content/uploads. I can not manually add folders under that directory unless I'm using the WP plugin Media Folders which can do that. It can however further complicate the upload process, because I technically can upload many files selected in bulk in one process, however, if I need to upload them into special folders created to hold them, I must later separate them into those folders manually or upload one by one to those folders. Yes, the structured directory navigation may work if probably I'm using this WP plugin, however, I do not know how that WP Media Folders plugin works I have not tried it yet and I do not intend to use it if there is a simpler resolution.

This entails and regards to;

Flattening the export structure if that is possible to hold the exported files in a single directory, and also maintain the left side navigation simply referring to those HTML files not containing the directory. example Vault (directories) --Organizations --- Global_Research (files) --Persons ---Peter_Koenig (files) I want to export files selected from both directories but I want to hold them in a single directory and use a left-side navigation that refers only to those files e.g. Global_Research.html (not Organization/Global_Research.htm).

The second use case for why I do not intend to create many exported folders;

Let you consider I am writing a book1, book2, both are structured and contain many chapters. In my vault I can hold more separated directory structures Vault (directories) --Books ---Book1 (directory) ----Chapter1 (directory) -----Chapter1.md (containing a heading structure from H1-H3) ----Chapter2 (directory) -----Chapter2.md (containing a heading structure from H1-H3) ... ----ChapterN (directory) -----ChapterN.md (containing a heading structure from H1-H3) ---Book2 (directory) ----Chapter1 (directory) -----Chapter1.md (containing a heading structure from H1-H3) ----Chapter2 (directory) -----Chapter2.md (containing a heading structure from H1-H3) ... ----ChapterN (directory) -----ChapterN.md (containing a heading structure from H1-H3)

The more chapters the more directories are created. What is not a good strategy I think. However, the left navigation may work if your plugin allows it to store the exported files in a single directory and refer to the chapter files without director references. Also, consider that is not a good strategy to create so many separate directories at the final WP site.

I may choose to assemble the Book1's Chapters into a single Book1.md file with Obsidian embed, however, there is yet a problem when exporting to HTML with the inconsistent TOC navigation when embed. (see https://github.com/KosmosisDire/obsidian-webpage-export/issues/386#issuecomment-2009550485) BTW: regarding this please consider why the transclusions need not come under the previous heading. On the contrary for example in an assembled bookcase into a single file, what embeds the Chapters files the heading hierarchy at right TOC much rather should look.

Books1.md (assembled with embedded chapters) Chapter 1 (H1) --Chapter 1.1 (H2) ---Chapter 1.1.1 (H3) ---Chapter 1.1.2 (H3) Chapter 2 (H1) (transclusion from Chapter2.md) --Chapter 2.1 (H2) ---Chapter 2.1.1 (H3) ---Chapter 2.1.2 (H3) ... Chapter N (H1) (transclusion from ChapterNmd) --Chapter N.1 (H2) ---Chapter N.1.1 (H3) ---Chapter N.1.2 (H3)

Also the file size for what the original issue refers very important when uploaded and truly slows down the site's performance. (I understand the packaging fonts are in beta why are the files so big...see my comment regarding this below)

[vgyenge6]

https://github.com/KosmosisDire/obsidian-webpage-export/issues/415#issuecomment-2026048821

Okay I know that I'm using the beta version :)

I would like to ask for. Has the final 1.8.0 this large file size issue? Has the final 1.8.0 the Make TOC links relative #366? Has the final 1.8.0 Add style support for Canvas Style Menu and Colored Tags Wrangler #384?

If not, unfortunately, I must wait to upload to WP anything until you finish this issue. :( (Note: The web export method separating the components of the exported HTML, css, scripts fonts, etc which decreases the file size as you say https://github.com/KosmosisDire/obsidian-webpage-export/issues/415#issuecomment-2023843526, probably I must try that yet.)

Thank you for reading my longer comment above and this 👍

[KosmosisDire]

Yes, the structured directory navigation may work if probably I'm using this WP plugin, however, I do not know how that WP Media Folders plugin works I have not tried it yet and I do not intend to use it if there is a simpler resolution.

As of right now there is not a simpler resolution other than to wait. You could try using the plugin's render API from the developer terminal in obsidian which would allow you to set specific render options including flattening the export. However this is very much a work on progress so I can't say anything about whether it would work, and it will also be a huge pain to work with since you have to write custom JavaScript just to export.

It is expected that uploading self contained documents to a web server should be slow. In fact the description of this mode explicitly states that this is a downside. So yes, you should be using the web server mode.

Your options right now are to wait, or try and use the media folder wp plug-in.

[KosmosisDire]

Have you tried connecting to wordpress via ftp and uploading your files via ftp?

[vgyenge6]

Thank you for your answer and suggestions.

1. I will try how the web export mode works. I have not tried that yet. I must try how can upload those separated files to WP and whether in this case the HTML in the iframe will be rendered and my scripts work correctly.

2. It has not been quite clear yet to me whether the 'flattening' is related to HTML export only in both web-export modes. I surely need to try step 1. before. to see that. But I guess now both.

You could try using the plugin's render API from the developer terminal in obsidian which would allow you to set specific render options including flattening the export. However this is very much a work on progress so I can't say anything about whether it would work...

Does this mean I must write my own export script? (It seems sure I won't personally enter into doing this :))

this is very much a work on progress

Does this mean you are considering working on resolving 'flattening' in the future?

Have you tried connecting to wordpress via ftp and uploading your files via ftp?

This is possible, yes, I can and a good idea given by you. Thanks.

But, I will make an effort to further tend to not use the many separated directory logic and another WP plugin. I will figure out what to do.
Regarding the above book-case exporting structure it seems better to assemble the files into one document embedding the chapters, but in this case, I must wait for how you resolve the transclusion heading structure in TOC. (see https://github.com/KosmosisDire/obsidian-webpage-export/issues/386#issuecomment-2009550485) BTW: regarding this please consider why the transclusions need not come under the previous heading. I will take a new comment referring to this on that issue topic. https://github.com/KosmosisDire/obsidian-webpage-export/issues/386#issuecomment-2028017204

Yes, in conclusion, I must wait for your work progress and try now what I can. :( :)

[KosmosisDire]

Flattening is already supported, but it is not part of the UI. (The files are flattened when only a single document is exported for example). I do plan to make this option available in the future through the UI.

[vgyenge6]

I do plan to make this option available in the future through the UI. It is welcome! And wanted waited,

I'm waiting for your plugin to be full-fledged. It is truly a gem polishing obsidian :D

[vgyenge6]

Hi,

I will try how the web export mode works.

It has not been quite clear yet to me whether the 'flattening' is related to HTML export only in both web-export modes.

I tried both. I can see how the exported folder structure and its content will look in both cases. Also how the components will be separated when web export mode is used. I tried to upload the 'web export' folder structure to WP by FTP. I also tried to install WP Media Library folders to handle this structure from the WP interface. Although this structure can be truly uploaded relatively simply by FTP, there are complications to simply using this structure from WP UI. (There are more WP Media Library folders to use but seems all have to handle a seemingly real file structure and getting the links simpler inside WP only if the paid PRO version is used.)

Furthermore, as I had written above my main concern is, I would like avoiding the many structured directories as it was in my vault and keep the left navigation as simple and variable as I would like to assemble the files. Even if the HTML export also will keep yet the directory structure until the 'flattenng is not developed fully' it is flexible for me to rearrange the files by hand into a single directory and repair the links - aka manually flattening the files before upoad to WP.

In conclusion I want to use the HTML export mode and thus the self-embedded version of them because it is much simpler to handle on WP .

I understand the beta issue regarding the font packaging, but if you allow the HTML export to local computers; The smaller file size stored on local computers is also very important. The increased file size because of this issue is 10x, which is very large.

Please help me, how can I eliminate the packaged fonts manually (decreasing the file sizes) from the HTML file until this issue is resolved? Albeit, I tried to open these large HTML files either with vscode or Notepad++ to see how to do it, but both app slowed down and I got performance issue on my local computer too.

I'm truly not impatient waiting for this issue resolved, however, it would be important for me to keep my work in sync because there are some HTML files that are single files (again, I want to use the self-embedded version) and I would like to upload those to WP in the meantime.

Could at least the 'Make TOC links relative' https://github.com/KosmosisDire/obsidian-webpage-export/issues/366? added to a version or which version do you suggest to use which does not generate such a large HTML exported file? even if that does not contain the extras what I would like to use. https://github.com/KosmosisDire/obsidian-webpage-export/issues/415#issuecomment-2027105994

I would like to ask for again. Has the final 1.8.0 this large file size issue when HTML export is used? Has the final 1.8.0 'Custom head content' feature at least?

Thank you for your kind understanding and answering me these questions to help my work to continue.

[KosmosisDire]

I don't remember exactly which version I added this in, but I think it is the same version that I added the relative TOC links.

I understand it is important to you, however your use case is very very niche. It is likely that I will resolve all these issues eventually but it may be a few versions.

There is not a way to manually remove the fonts easily. Again you could use the export API and write a custom script to export your files but that's basically the only option other than to wait.

[vgyenge6]

Hi,

your use case is very very niche. It is likely that I will resolve all these issues eventually

I think my use case for trying to use your plugin capabilities is truly a gap in using Obsidian Notes on WordPress. As I mentioned web export absolutely won't work on WP because it does not have a real file structure thus the links to components to render the HTML file also won't work. Only the self-embedded HTML export scenario can be used but the files are truly very big to upload even if I only want to export a single note this way.

Meanwhile, I regressed to 1.8.01 final which can not make such big files and contains the Custom head content capability at least what I can use in a single note HTML file export. However, the relative TOC links are very missing for me, (even if I only want to assemble files selected from the same directory both left and right navigation worked well), and also rendering the web page put on canvas what has already worked in the 1.8.1-2beta too.

Anyway, I thank you for considering fully developing those issues we touched on. I will be waiting for that.