search

gohugoio / hugo

The world’s fastest framework for building websites.
https://gohugo.io
Apache License 2.0
75.76k stars 7.53k forks source link

Non-MD files in the content directory (like jpg's and png's) end up as html's #147

Closed RubenN closed 9 years ago

RubenN commented 10 years ago

When I put non-md files in the content directory, they should just be copied over (untouched!).

Now they end up as html's

spf13 commented 10 years ago

This is a bug, but I don't think you'll want the fixed behavior either.

/content is for documents that need processing /static is for content that doesn't need processing

I think the fix here should be to ignore any filetype hugo doesn't know how to process.

Does this make sense? Is there a reason for putting non parsed files like pngs in content vs static?

RubenN commented 10 years ago

I would say that content should be what the term says: a collection of info like documents, pictures, etc. Static should be for the interchangable parts like layouting, css, etc.

The most important reason is to keep content together (pictures in a blog for example).

I agree that ignoring is the most sensible thing to do!

2013/12/3 Steve Francia notifications@github.com

This is a bug, but I don't think you'll want the fixed behavior either.

/content is for documents that need processing /static is for content that doesn't need processing

I think the fix here should be to ignore any filetype hugo doesn't know how to process.

Does this make sense? Is there a reason for putting non parsed files like pngs in content vs static?

— Reply to this email directly or view it on GitHubhttps://github.com/spf13/hugo/issues/147#issuecomment-29737091 .

robotamer commented 10 years ago

I think the way content is interpreted here is: Files that need processing. Static files are js, css but also image, video, pdf To keep organized you could do something like
/static/image/blog/...

robotamer commented 10 years ago

Also there is one more interesting thing.

There are is a static folder and it has a sub folder called static as well.

like this: static/static/js
if you place a static file in to the first static folder it shows up in the root of the public folder. Haven't tested this but you could create a folder in there as well.

looking like this `static/blog/image/foo.jpeg

Then your public folder will be organized the way you are asking for..

spf13 commented 10 years ago

Correct.

Best, Steve

--  Steve Francia http://stevefrancia.com http://spf13.com http://twitter.com/spf13

On December 4, 2013 at 2:23:10 PM, Dennis T Kaplan (notifications@github.com) wrote:

Also there is one more interesting thing.

There are is a static folder and it has a sub folder called static as well.

like this: static/static/js

if you place a static file in to the first static folder it shows up in the root of the public folder. Haven't tested this but you could create a folder in there as well.

looking like this `static/blog/image/foo.jpeg

Then your public folder will be organized the way you are asking for..

— Reply to this email directly or view it on GitHub.

luckyrandom commented 10 years ago

Is there any update on this issue? It's inconvenient to put all images in separate directory, especially when plots are generated by scripts supplementing the post.

On the other hand, care must be taken when images or other assets are put in local directory. It may encourage the use of relative path, which may link to the wrong path when partial rendering is included by paginations or other pages.

I checked a few popular static site generator. wintersmith seems to be the only one to support local assets and relative links. It patches markdown renderer to support that, and it may not be a easy job if multiple renderers need to be patched.

cburgdorf commented 10 years ago

I think separating strictly by file type is backwards. If I have a picture only for one post, I want it to be next to my post.md file.

spf13 commented 10 years ago

I see your point, but I'd like to clarify the intent here.

We're not separating by file type, but by purpose.

/Content is meant to be compiled by hugo.

/static will be copied without any processing

HTML and md are welcome in either location. If in /static they will be copied without being touched.

I'm willing to concede that this separation may not be necessary and perhaps determining intent only by file type would be appropriate, but it is more limited and would make the watching capability less efficient and more complicated.

Steve Francia spf13.com @spf13

On Apr 11, 2014, at 4:18 PM, Christoph Burgdorf notifications@github.com wrote:

I think separating strictly by file type is backwards. If I have a picture only for one post, I want it to be next to my post.md file.

— Reply to this email directly or view it on GitHub.

cburgdorf commented 10 years ago

Understood. Would still like to have that feature :) I'll investigate the possibilities/impact down the road once I got my feet wet with the code base. Still struggling to get my environment set up.

jcw commented 10 years ago

Splitting files by how Hugo treats them is a bit of a software-centric approach, to be honest. When adding a blog post, it really is more convenient to create the post and everything it needs in one place. Even systems which split things up (WordPress, MarsEdit) have moved towards making it easy to do everything in one spot (usually via drag and drop, which is not feasible with Hugo, since it doesn't deal with the editing process itself).

I vote a big +1 to support images next to the text of a post/article. This should be all about the workflow of people, not how Hugo sees things.

spf13 commented 10 years ago

@jcw I think that's a pretty compelling argument. Let's plan on improving this for v0.12.

RubenN commented 10 years ago

Excellent, I couldn't agree more!

Op maandag 19 mei 2014 heeft Steve Francia notifications@github.com het volgende geschreven:

@jcw https://github.com/jcw I think that's a pretty compelling argument. Let's plan on improving this for v0.12.

— Reply to this email directly or view it on GitHubhttps://github.com/spf13/hugo/issues/147#issuecomment-43458988 .

jcw commented 10 years ago

Thanks for considering this, Steve. I'm really looking forward to be able to use this - would like to convert a large weblog filled with images over to hugo. The combination of markdown, speed, and flexibility (plus the fact that you're actively developing it further) makes it a no-brainer for writing just about anything for the web, IMO.

oscar-b commented 10 years ago

@spf13 Just wanted to give my +1 to this as well. I started looking at migrating a clients portfolio site to Hugo (currently in ZenPhoto), but I quickly realized that there is no way to dynamically resize images. I would want to put the original images in albums, and being able to generate thumbnails and appropriate sized versions from the originals (think <picture></picture> with @2x and @3x versions).

In this context (and your own roadmap which lists "Dynamic image resizing via shortcodes"), content images should go to /content, since they need processing by Hugo :)

mohae commented 10 years ago

Some thoughts on this.

Support a per fileType processor that would process content files based on their file extension. If a processor for that fileType is not implemented, the file is not processed.

This would allow for current functionality, in regards to .md files, support other markup languages by allowing processors to be added for them, and various image processing, when implemented.

I'm sure I'm missing some obvious stuff here. Any thoughts?

stou commented 10 years ago

I aggree with @jcw that separating content text, and content images and other media in separate folders feels awkward.

Maybe someting like the processing chain used by docpad could be useful: https://docpad.org/docs/begin#a-note-on-rendering

Just an idea

aledalgrande commented 10 years ago

+1 for separating by context and not application/processing based

spf13 commented 10 years ago

@stou ,

I've done some work on this already and there's a bit more to do still. I'm a bit confused about your comment about docpad. Docpad takes the exact same approach as Hugo does now by placing all binary files (images) into the /static directory.

I understand the chaining abstraction they provide, but to me that seems like a novelty. Why would anyone what to process files from one markup into another into another? (md/asciidoc/??) -> html seems sufficient.

stou commented 10 years ago

@spf13 About the chaining I think I got mixed up with some of the comment in issue #320

I still think the chaining could be a usefull way to configure processing of files e.g.:

file.html.md --> file.html
file.js.coffee --> file.js
file.html.jade --> file.html
file.xml.jade --> file.xml
file.css.scss --> file.css

if you add more preprocessors in the future. Consider this a feature request, but maybe the file extension idea could be used as part of the solution to issue #320 instead of adding a lot of configuration options / extra settings in front matter.

kot-behemoth commented 9 years ago

Any updates on this? Being able to have each post contain all the assets (be it images or scripts) next to the post's text would be invaluable. I'm thinking of moving to Hugo, so had a look at the docs, but couldn't find a confirmation of being able to do this just yet.

Thanks!

mrkishi commented 9 years ago

Any updates on the custom processors, as well? It's an idea that also interests me. For instance, one could optimize images on build.

Honestly, I found the content folder somewhat misleading as it's not meant to hold arbitrary site content, but processable inputs.

Also, if we are to think them as raw, unprocessed documents, it makes sense that's the place we'd place related data, full resolution images and etc., as well. They would then be processed on build: charts generated, images resized and optimized, html generated. Of course, a pass-through processor is still a valid processor. :)

As not every file could (easily) have a front matter, @stou's file extensions idea seems like an acceptable solution for processor selection, if slightly magic. Some ideas of what may be possible:

spf13 commented 9 years ago

I like this idea, but @stou's file extension idea only allows for a single result from the process. What if I want to make a thumbnail, medium size and full size from a single image?

mrkishi commented 9 years ago

How about introducing the concept of external alternatives to front matter? Appendix, lol?

This could make things slightly more consistent: every input (to be processed) must have a front matter, either embedded (markdown's front matter) or external (stand alone yaml and toml files that select the processor?). On the other hand, it adds to the project complexitivity and, potentially, build time.

This could get annoying pretty fast, though. Another option would be to add the processing options to the pages' front matter:

I think this could work, but it would limit our ability to process arbitrary files: they would have to "belong" to an article. Perhaps on top of that there could exist support for stand alone .toml and .yaml files, that simply contain instructions for other files, similarly to the first suggestion but without the 1:1 toml/file relation.

bguiz commented 9 years ago

Think we might be getting a little side tracked here, with the discussion on preprocessing files. That's a nice to have for non markdown files, but for a first cut, I will be quite happy in *.md files are the only ones that are processed, and everything else just gets copied as-is.

Creating a rails or docpad style asset pipeline based on multiple file extensions is a nice to have, but I think simply being able to put images referenced in a markdown file in the same folder as the markdown file will aid authoring in Hugo quite a lot.

jcw commented 9 years ago

+1

bep commented 9 years ago

+1 what bguiz says

stou commented 9 years ago

+1 I agree. The processing chain can easily be postponed

bep commented 9 years ago

One note to he who implements the "copy files from content without processing" -- it would be very nice if static images/css could be treated as their cousins in the /static folder for livereloads (soft, not hard refreshes).

spf13 commented 9 years ago

I'm 90% done with this today.

Steve Francia spf13.com @spf13

On Jan 23, 2015, at 11:02 AM, Bjørn Erik Pedersen notifications@github.com wrote:

One note to he who implements the "copy files from content without processing" -- it would be very nice if static images/css could be treated as their cousins in the /static folder for livereloads (soft, not hard refreshes).

— Reply to this email directly or view it on GitHub.

dcorb commented 9 years ago

Glad to arrive to this issue and it's 90% ready! This improves the UX of writing complex articles.

I was thinking to have a developer blog with "demo.js" or "demo.css" per article or even JSON files .. so each article could be customized. Not only it would be useful for dev blogs.. also is a trend called "JavaScript Journalism", like you can see in some New York Times articles.

gima commented 9 years ago

+1 The current logic feels counter-intuitive. Some posts need assets, such as images. These images are, by their name, assets, and hence belong with the post. I shouldn't have to worry about where they are located as a result of rendering the website, but currently I have to. I have to either refer to them by their rendered path or use the static folder trick to mirror the post's directory structure, and place my assets there, afterwhich I can then refer to them in the post simply by their name.

This is awkward. Not only for the extra work, but also for the fact that the filesystem already provides for a way for me to bundle things together that belong, together, by using directories. Now if I have to mirror this directory structure, I have broken the connection between the post and it's assets [in the filesystem], then during rendering Hugo rebuilds the connection, and as a result the outcome is the same as if Hugo just let me have my damn files in the post's directory and be done with it.

spf13 commented 9 years ago

100% agree. Working on making it better.

yml commented 9 years ago

@spf13 are you pushing your work somewhere ? I would be very interested to see this being built. I have been working myself on thumbnailing pipeline recently and would be willing to contribute / adapt some of it to hugo.

ivanjuras commented 9 years ago

+1

spf13 commented 9 years ago

It's in master. The foundation is there to handle specific file types differently. CSV already has a handler as an example / prototype.

Steve Francia http://stevefrancia.com http://spf13.com http://twitter.com/spf13

On Wed, Mar 4, 2015 at 5:38 AM, Ivan Juras notifications@github.com wrote:

+1

— Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/147#issuecomment-77132928.

marcojakob commented 9 years ago

+1 It would be awesome to be able to store assets next to the content files.

As I've outlined in File-Based CMS and Hugo as Static Site Generator it would be great to also allow .toml files as content instead of .md files.

eduncan911 commented 9 years ago

@spf13 You stated the foundation is already in master. Could we get an example of how to use it? I'm currently getting the errors like this:

Hugo Static Site Generator v0.14-DEV-BE01F84 BuildDate: 2015-04-20T08:25:34-04:00

CRITICAL: 2015/04/20 Errors reading pages: Error:No handler found for Dell_H2C_Technology_Hybrid_Cooling_for_Overclocked_CPUs.pdf
Error:No handler found for R201623_Silicon_Image_SIL3132_Controller.zip
Error:No handler found for Utility-Alienware_AlienFX-R202930.zip
Error:No handler found for Utility-Dell_XPS_Thermal_Monitor_A051.0.0.11.zip
Error:No handler found for bios_DellXps730xBios_A11b.zip
Error:No handler found for AlienFX_Lights.png
Error:No handler found for MCBBinaryMap-20100203.xbin

Out of all comments, @gima said it best: it is akward and the file system already provides me a way to organize with directories. :+1:

My current blog, started way back in 2002, has been through 3 serious "technology/generator" changes. In doing so, and preventing broken links, I have a wide swath of existing static content next to posts and articles. Things like:

/blog/archives/2015-02-22-hello-world.aspx
/blog/archives/images/hello-world.jpg
/blog/2nd-overhaul-posts.html
/blog/images/newer-images-and-thumbnails-here/...
/current-version-at-root.html
/images/different-format-images

And I have quite a few videos, PDFs and such lined up next to their posts.

I liked Octopress exactly for this, as I was free to create any directory and/or file I wanted anywhere, and it would be copied over in that exact place. I will admit that Hugo's forced separation approach irked me for some time - and still does. I haven't launched my new overhaul yet because of these restrictions - I don't want to separate 1000s of posts and content.

Asking me to break up my static site for a static migration is a deal breaker.

Octopress (aka Jekyll) allowed you to create your own structure - and it would only process files by their extension type. .html files are treated differently than .md files. Have a new .adoc type? No problem, copy a plugin (or write your own) to process that file extension. Have files with .gzip, .tar.gz, .zip, .7z, .7zip and .rar extensions? No problem, they are copied as-is and not processed/touched.

^- Hey, someone had to mention Octopress/Jekyll already does this in this thread (and kind of trains you to do this).

bguiz commented 9 years ago

@eduncan911 Yup, you're venting the very same frustrations that I have with Hugo, when I commented on this thread earlier: https://github.com/spf13/hugo/issues/147#issuecomment-71126819

Think we might be getting a little side tracked here, with the discussion on preprocessing files. That's a nice to have for non markdown files, but for a first cut, I will be quite happy in *.md files are the only ones that are processed, and everything else just gets copied as-is. Creating a rails or docpad style asset pipeline based on multiple file extensions is a nice to have, but I think simply being able to put images referenced in a markdown file in the same folder as the markdown file will aid authoring in Hugo quite a lot.

Rather than whitelist a select number of file extensions which should get pre-processed, and error on everything else, I would much rather have a select number of extensions which get pre-processed, and have everything else simply pass through, by simply getting copied from the source folder to the destination folder.

I am currently using Hugo on a couple of static sites, but am still stuck on Docpad for my blog. I would very much like to switch that to Hugo too, but I am waiting for this issue to pan out, mainly due to me not wanting to copy images that are currently in the same folder as my markdown files into a separate parallel folder structure - it would simply be way too much tedious work!

@spf13 could we please get files extensions without a handler simply copied across?

spf13 commented 9 years ago

Yes. I'll work on that this week.

bep commented 9 years ago

"I'm 90% done with this today."

This was January. I understand it's not uncommon in software development that the last 10% is the hardest - but is it possible to get a realistic estimate? Should we remove this from the v.0.14 milestone?

bguiz commented 9 years ago

I'd really really like to see this feature make it into the next release, can we please keep it in the milestone?

W: http://bguiz.com

On 30 April 2015 at 04:22, Bjørn Erik Pedersen notifications@github.com wrote:

"I'm 90% done with this today."

This was January. I understand it's not uncommon in software development that the last 10% is the toughest - but is it possible to get a realistic estimate? Should we remove this from the v.0.14 milestone?

— Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/147#issuecomment-97531420.

bep commented 9 years ago

v0.14 is long overdue and we cannot release something that isn't there, but I'm not doing the release, so I wouldn't know.

jbenet commented 9 years ago

just ran into this problem. it's really annoying, particularly when using hugo as part of a pipeline for a bigger site.

gour commented 9 years ago

Let me just add that the other day I was playing with Nikola which allows to use code like this:

/path-to/content/post/hello-world.rst
/path-to/content/post/hello-world/image.jpg

and the image is referenced within the post with:

.. thumbnail:: image.jpg
                  :alt: Some alt-text
                  :align: center
                  :scale: 60        
                  etc.

Simple & clean way to keep images/assets along with posts...

eduncan911 commented 9 years ago

Hello @Gour

Could you give us more details?

What version were you using? Latest from master?

Can you give/link to a full example? I don't recognize :alt: :scale:.

Thanks! On May 7, 2015 1:09 AM, "Gour" notifications@github.com wrote:

Let me just add that the other day I was playing with Nikola https://getnikola.com/ which allows to use code like this:

/path-to/content/post/hello-world.rst /path-to/content/post/hello-world/image.jpg

and the image is referenced within the post with:

.. thumbnail:: image.jpg :alt: Some alt-text :align: center :scale: 60 etc.

Simple & clean way to keep images/assets along with posts...

— Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/147#issuecomment-99720985.

gour commented 9 years ago

@eduncan911 I am speaking how one can tie one's assets (e.g. images) close to the post in Nikola static-site-generator, iow. it's not about Hugo, but example how it could the problem of tying images close to the post be solved within Hugo.

eduncan911 commented 9 years ago

Oh, I didn't catch that. Yeah, Octopress/Jekyll does as well.

On Thu, May 7, 2015 at 3:14 PM, Gour notifications@github.com wrote:

@eduncan911 https://github.com/eduncan911 I am speaking how one can tie one's assets (e.g. images) close to the post in Nikola https://getnikola.com/ static-site-generator, iow. it's not about Hugo, but example how it could the problem of tying images close to the post be solved within Hugo.

— Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/147#issuecomment-99986020.

derekperkins commented 9 years ago

This seems like a great place to hook in an image processing pipeline, which has two major stages I can think of:

This is obviously much more intensive than just generating html files, so it'd have to come with some sort of caching mechanism so it didn't have to copy and regenerate all the image files every single time.

derekperkins commented 9 years ago

The more I think about it, the more that seems a little intense for Hugo. While it would be super nice if it could fit the workflow, it may be more suited to a CI environment to go through and build all the right sizes on deployment. The only thing you lose by doing that is not being able to set the right width / height attributes inside your html tags.

eduncan911 commented 9 years ago

This is not to address only images next to md files, but all file types.

We all agree that Hugo:

1) should not try to process files it doesn't know about in the /content directory. 2) move all files it finds in /content over to that same directory in the generated site, keeping the exact same structure.

E.g., side-by-side files:

/content/post/summer-fun/what-i-did-last-summer.md /content/post/summer-fun/jet-skiing.png /content/post/summer-fun/jet-skiing.mkv /content/post/summer-fun/jet-ski-for-sale.zip

What you mention is another idea: file handlers. One could create their own handler and process the files with an external script or something if it matches an extension. But that's a different discussion for another thread. :)

On Thu, May 14, 2015 at 8:19 PM, Derek Perkins notifications@github.com wrote:

The more I think about it, the more that seems a little intense for Hugo. While it would be super nice if it could fit the workflow, it may be more suited to a CI environment to go through and build all the right sizes on deployment. The only thing you lose by doing that is not being able to set the right width / height attributes inside your html tags.

— Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/147#issuecomment-102207530.