parcel-bundler / parcel

The zero configuration build tool for the web. 📦🚀
https://parceljs.org
MIT License
43.5k stars 2.27k forks source link

Is there a way to import json as a static file? #501

Closed mcfarljw closed 4 years ago

mcfarljw commented 6 years ago

I'm using a few libraries that want to load json from a path. Is there a way to serve static files (similar to what the copy-webpack-plugin does)?

DeMoorJasper commented 6 years ago

require, import and fs.readFileSync will all include the json file into the bundle

mcfarljw commented 6 years ago

Hmm, @DeMoorJasper can you clarify a bit? I just tried using require for my json file but it still gives me an object rather than a static file path.

DeMoorJasper commented 6 years ago

That's the point, it resolves the json file as an object, it parses it for you.

mcfarljw commented 6 years ago

Sorry, I read that as use require.... import and fs.readFileSync will all include the json file into the bundle. The doesn't really help answer my question, but rather just confirms what I already know. Does that mean there is no way to import a json file as a static file rather than having it included in the bundle?

DeMoorJasper commented 6 years ago

The only way i can think of resolving json as a raw asset is to overwrite the asset type by writing a plugin

mcfarljw commented 6 years ago

Hmm, I wouldn't really want to overwrite an asset type. I guess what I'm looking for is a way to use parcel in development that works like the static-serve middleware for express.

devongovett commented 6 years ago

Why do you need to copy the JSON file instead of bundling it into your code? Just trying to understand the usecase.

mcfarljw commented 6 years ago

@devongovett I use a few 3rd party libraries that only support passing a path to load json files via xhr. The example I ran into today was with phaser 2 sprite atlases:

game.load.atlas('items', 'assets/sprites/items.png', 'assets/sprites/items.json')

It's probably not too common, but I can recall using a few other libraries that have their own preload systems that treat json assets the same as images.

sunnylqm commented 6 years ago

@devongovett For example, I want to lazy load some (or a lot) json data.

shunia commented 6 years ago

@sunnylqm Fair point. We run into this problem too as we have a bunch of assets(images,json files,etc.) to be loaded.

require & import does most the way we wanted, except with json files, it bundles json into javascript instead of return the path map. But what we wanted is the return the path way as other files do.

The reason is, we are building a loading sequence for all the assets other than javascript files, to show a loading bar to users, which will indicates the loading progress.

Webpack actually offers a extended require function WebpackRequire for us to make this happen, by mapping file path before bundle and after (WebpackRequire.context), we can dynamically replace path used inside javascript codes into hashed path, to make a successful load (by xhr) action.

devongovett commented 6 years ago

See here for some potential strategies to allowing multiple import formats: https://github.com/facebookincubator/create-react-app/issues/3722

FDiskas commented 6 years ago

you can try parcel-plugin-json or babel-plugin-inline-json-import

RELNO commented 6 years ago

To extend @shunia & @mcfarljw point, THREE - THREE.ObjectLoader() API is XHR JSON loader which fails in parcel dev. server:

screen shot 2018-04-14 at 21 34 02
FDiskas commented 6 years ago

https://github.com/parcel-bundler/parcel/pull/536

michaeljota commented 6 years ago

Just to say this is needed if you are using something like pixi.js. I understand not a huge user base, but is something. I guess I have to go back and use Webpack, and waste 4 hours of my life.

Or do: https://github.com/parcel-bundler/parcel/issues/1080#issuecomment-376823475

Thanks.

shunia commented 6 years ago

@michaeljota Change file type from .json to something like .data, then let pixi.js to load them as json type. Problem solved. But you may lose type check(json format validation) from your ide.

michaeljota commented 6 years ago

I just try that and I does not work.

The problem now, is exactly about the json atlas files. But not quite because is a json file or not.

1) pixi seems to validate the kind of file that you are trying to load, and I don't quite know how to tell pixi this is a json file with another name 2) The atlas file has a property that should resolve to the name of the atlas file. Because Parcel will always add a hash to whatever resource I need to load, the name in atlas will never match. (Again, if this is possible, please tell me how, because I really don't know how).

Having a bundler tool with zero config is great, but I guess you need to consider what are the main use cases you want to cover, and at the end, I think is real hard to cover them all.

shunia commented 6 years ago

@michaeljota It's not so straight forward to use neither parcel nor webpack, to work with pixi.js. Some extra work need to be done when using parcel, same as webpack.

If you want to get the real path after parcel builds your .json file(after changing your file type to .data for example), this would just works:

  let jsonURL = require('path/to/your/json/file');

If you want to load any file as json format file(of course they need to be parsed correctly by JSON.parse), just provide a proper loaderType/XHRType to pixi.js's loader, something like:

  PIXI.loader.add({
    url: jsonURL, 
    loader: 'json loader type defined by pixi.js', 
    xhrType: 'json xhrType defined by pixi.js'
  });

If you want to patch json data to atlas resolver, you have to understand how pixi.js's resource loader is working, and write some hack code to make it working, I can not provide example here.

michaeljota commented 6 years ago

If you want to get the real path after parcel builds your .json file(after changing your file type to .data for example), this would just works:

 let jsonURL = require('path/to/your/json/file');

I do it that way, and this is the way I use to load things that are not a json file. This is the main issue with this approach. The JSON file have a property that points to the path out the atlas image. As I understand there is not currently a way to tell parcel to move a file without renaming it, so there is no way for the JSON atlas describer to know where the atlas image will be after the build.

If you want to load any file as json format file(of course they need to be parsed correctly by JSON.parse), just provide a proper loaderType/XHRType to pixi.js's loader, something like:

I don't know much about pixi but I did lookup for something like this, and the loader, and xhrType options seems to be related more to something like xrh, audio, video, and image. But I did not found something like json.

jedhastwell commented 6 years ago

This is not strictly answering the question, but for those of you just looking to load a PIXI sprite sheet, you can bypass the PIXI loader and manually create the sprite sheet from the raw data:

  const data = require('../assets/atlas.json');
  PIXI.loader.add('atlas', require('../assets/atlas.png'), (resource) => {
    if(resource.error) {
      console.error(resource.error);
    } else {
      const texture = resource.texture.baseTexture;
      const sheet = new PIXI.Spritesheet(texture, data);
      sheet.parse((textures) => {});
    }
  });
michaeljota commented 6 years ago

@JedHastwell This will create an atlas key in the TextureCache with the data from the atlas descriptor or something like that? I'm sorry, I just don't understand Pixi that well.

jedhastwell commented 6 years ago

@michaeljota It's the same as if you'd just used the loader to load the atlas file directly from a JSON file. It will add all the textures defined in the atlas file to the TextureCache. Then you can just create sprites using the names defined in the atlas like: let ball = Sprite.fromImage('ball.png');

michaeljota commented 6 years ago

Oh! OK, OK... Will try that. Thanks you.

@JedHastwell Thanks man. Really helpful. Worked as intended. Thanks again.

skozin commented 6 years ago

To extend the point for adding static files serving/copying: I'm using some proprietary third-party library that expects me to pass URL to a static directory provided with the library. This directory contains about two hundreds of files, something that looks like this:

image

These files are JS/CSS/HTML files, as well as images, cursors and fonts. I cannot require every single file, and even if I could it won't help as the library expects to get each of them using XHR under specific, compiled-in name. I can only change shared URL prefix.

As far as I can see, in order to make this setup work under Parcel, I need two things:

  1. When building production bundle: ability to copy a bunch of files and folders to the destination directory without any parsing and modification.
  2. When developing: ability to serve static contents from a local directory under specific URL prefix.

Please let me know if this can be accomplished in some other way.

shunia commented 6 years ago

@skozin Just try const assets = require('relative/path/to/assets/folder/*'). Notice the * in the url. The object assets should contains everything you need.

michaeljota commented 6 years ago

As I understand, the problem is not loading those files, but letting those files load themselves.

skozin commented 6 years ago

@shunia, the problem is not loading those files, as @michaeljota correctly said. I don't need them; the library I'm using needs them, and it wants to load them using XHR, and expects them to have pre-defined file names which are compiled into the library code. It does this to avoid loading unnecessary stuff: the total size of all these files is something like 20Mb, and most of them are not needed in every case. I suppose that .js bundles were actually generated using webpack's code splitting mechanism and are being loaded using dynamic imports or require.ensure. Note that I have no access to the library sources, I can only use already bundled and minified code.

shunia commented 6 years ago

Try parcel-plugin-json-url-loader

stevage commented 6 years ago

Just wanted to add my use case: loading Geojson files (that is, spatial data containing location information). It doesn't make any sense to me to bundle that data. It would make the bundle enormous, and presumably all the loading would have to happen before the site displayed. (In my case I'm using mapbox-gl-js, which means I could pass the loaded geojson object, instead of a URL...but I'd prefer not to.)

Really surprised there isn't a simple solution like, "anything in /static doesn't get bundled" and can be loaded using XHR.

kpollich commented 6 years ago

+1 for the aforementioned three.js use cases here. Looking into migrating a WebGL project away from Webpack, but we need to be able to load object files and other WebGL assets via XHR.

skozin commented 6 years ago

@shunia thanks! we'll try this :)

mikedpad commented 6 years ago

Just started using Parcel recently (❤️it, btw) and ran into this issue recently myself. My use case was much simpler... I just wanted to include (copy) some static HTML files over to the dist folder.

I eventually just used an anchor in my index.html entry to include the file:

<body>
  <noscript>You must enable JavaScript to use this app.</noscript>
  <div id="root"></div>
  <a href="myStaticFile.html"></a>
  <script src="./index.js"></script>
</body>

</html>

... which is incredibly hacky. I don't want to link from the page, but this is how I got them to be included without bundling them.

Like @stevage mentioned, it'd be great if there was a simple /static subfolder where assets aren't processed or bundled.

NotIntMan commented 6 years ago

@shunia, parcel-plugin-json-url-loader works bad when you need to refer json from html.

<link rel="manifest" href="~/../static/manifest.json">
shunia commented 6 years ago

@NotIntMan This plugin is not intend to do this. Because this intention has been handled by parcel. Try use relative path to your html file rather than a absolute path to your computer!

thejohnfreeman commented 6 years ago

My use case is I want a separate JSON configuration file that my administrator can tweak in the deployment. (I would love to use continuous integration, but I have to fit my solution into the problem's constraints.)

kleinfreund commented 5 years ago

How do I get Parcel to resolve a path like this and add the JSON to the output?

xhr. open('GET', '../data/some.json', true);

This flat out doesn’t do anything expected. For some reason, the result is not even a 404 error, but my HTML entry point file. This is extremely confusing.


Weird workaround:

npm i parcel-plugin-json-url-loader --save-dev
xhr. open('GET', require('../data/some.json'), true);
bsides commented 5 years ago

Just wanted to leave my use case here.

I'm developing a Chrome Extension and it is required in the final build folder to have a manifest.json file there. Right now I'm just cp manifest.json ./dist/ but I wish I could just put it as an entry file with an option to just copy it.

connorjclark commented 5 years ago

A use case for bundling JSON as a string and deferring parsing to runtime: it can actually be faster. The JSON grammar is faster to parse than JS.

puka-tchou commented 5 years ago

For all of you who just need to copy a file from a static directory to the bundled one, I would recommend giving a shot at parcel-plugin-static-files-copy. I don't know how it handles the paths though.

bmmpt commented 5 years ago

Seems like everybody is adding their usage case here, so why not? :)

I am beginning to use the amazing react-i18next library and it's ability to asynchronously load new translations and add them to the application context. It supports plugins for xhr and fetch, but in both cases I get the HTML entry point file.

Both plugins try to fetch: http://localhost:3005/locales/en-CA/translation.json

I don't even have to tell i18next where to get the files from as long as I put them in the default path it looks for translation files (public/locales/...). I also have no say on how it's getting the translations, neither would I want to. That's the beauty of the plugin. That being said, I can't even use the "weird workaround" with require mentioned above.

There will be dozens of use cases for stuff like this. Can Parcel be awesome at this too?

mischnic commented 4 years ago

With Parcel 2 alpha 3 and this parcelrc

{
  "extends": "@parcel/config-default",
  "transforms": {
    "url:*": ["@parcel/transformer-raw"]
  }
}

you can import files which will then be copied to the dist folder:

import file from "url:./static.json"
fetch(file)...
deepred5 commented 4 years ago

use pixi.js and load a json file . Thanks @jedhastwell

import { Application,  Sprite, Loader, Spritesheet } from 'pixi.js';

import myjosn from './assets/treasureHunter.json';
import mypng from './assets/treasureHunter.png';

const loader = Loader.shared;

const app = new Application({
  width: 300,
  height: 300,
  antialias: true,
  transparent: false,
  resolution: 1,
  backgroundColor: 0x1d9ce0
});

document.body.appendChild(app.view);

loader
.add('mypng', mypng)
.load(setup)

function setup() {

  const texture = loader.resources["mypng"].texture.baseTexture;
  const sheet = new Spritesheet(texture, myjosn);
  sheet.parse((textures) => {
    const treasure = new Sprite(textures["treasure.png"]);
    treasure.position.set(100, 100);
    app.stage.addChild(treasure);
  });
}
danmarshall commented 4 years ago

Try parcel-plugin-json-url-loader

This worked for me, thanks @shunia ! 🥇

n8isjack commented 4 years ago

So I am using Parcel... love it and don't want to go back to webpack.

I have been dealing with this and my use case is for Pixi.js with spritesheets. I can't get it to work without a static file name. (I have done d3 maps before and see how it would be the same problem)

I also have cases where I want the JSON loaded as an object, so I can't just over-ride JSON imports.

In the end I was able to get PIXI to work with a data url.

I suppose if it was a huge file I'd have to learn about Parcel's Code Splitting for lazy loading.

import sprite_png from "../sprites/sprites.png";
import sprite_data from "../sprites/sprites.json";

sprite_data.meta.image = sprite_png;

let data64 = btoa(JSON.stringify(sprite_data));
let dataURL = `data:text/json;base64,${data64}`;

const theGame = new PIXI.Application({
  resizeTo: window
});

theGame.loader.add("sprites", dataURL);
wyozi commented 4 years ago

For playcanvas users struggling with .json model imports, you can rename model .json files to .modeljson and add a custom loader:

(app.loader.getHandler("model") as pc.ModelHandler).addParser(
  new (pc as any).JsonModelParser(app.graphicsDevice),
  function (url, data) {
    return pc.path.getExtension(url) === ".modeljson";
  }
);
opyate commented 4 years ago

Just leaving my workaround here for Phaser 3 + Parcel for the simple case where there's one spritesheet, in case it helps anyone.

ls -1 assets
other.png
spritesheet.json
spritesheet.png

import Phaser from 'phaser'
// @ts-ignore
import images from '../assets/*.png'
import spritesheetJson from "../assets/spritesheet.json"

export default class BootScene extends Phaser.Scene {
  constructor () {
    super({ key: 'boot' })
  }

  preload () {

    console.table(images)

    this.load.image('other', images.other)

    spritesheetJson.textures[0].image = images.spritesheet.substring(1)
    this.load.multiatlas('mysprites', spritesheetJson)
  }

  update () {
    this.scene.start('next-scene')
  }
}

Elsewhere:

this.add.sprite(0, 0, 'mysprites', 'the-key')
TimDaub commented 3 years ago

With Parcel 2 alpha 3 and this parcelrc

{
  "extends": "@parcel/config-default",
  "transforms": {
    "url:*": ["@parcel/transformer-raw"]
  }
}

you can import files which will then be copied to the dist folder:

import file from "url:./static.json"
fetch(file)...

It seems transforms was renamed to transformers: https://v2.parceljs.org/configuration/plugin-configuration/

TimDaub commented 3 years ago

A few days ago, I tried importing a .json file in the source code that I manage using parcel. My assumption is that anything that I use in an import ... from ... from statement gets bundled into my regular bundle. I'm aware that *.json imports are not yet stable in the current ES specification. That's OK.

I then discovered this thread and saw that different from my expectation, many people want to lazy load *.json files to optimize for their user experience. Here's a few of those:

However, the problem is that I think the import ... from ... statement was never intended to lazy load anything. That's what import(...) is for. Hence, I find it to be a regression that we now can define import paths starting with url:... for import ... from "url:..." to code split and lazy load (as I pointed that out in my last comment).

From parcel, what I'd expect when inputting import ... from "*.json" is that it bundles the *.json file in the main bundle once the ES standard on `.json imports has matured.

To import JSON with import ... from ..., I recommend using babel-plugin-inline-json-import. You can find instructions on how to integrate on the npm page.

jaredkrinke commented 10 months ago

With Parcel 2 alpha 3 and this parcelrc

{
  "extends": "@parcel/config-default",
  "transforms": {
    "url:*": ["@parcel/transformer-raw"]
  }
}

you can import files which will then be copied to the dist folder:

import file from "url:./static.json"
fetch(file)...

Thanks for this! The only tweak I would suggest is to use URL since it's more obvious that it's producing a URL (or string). I was able to use this for lazy loading translations for react-intl:

const urlToFetchForJson = new URL("url:../content/messages-compiled/zh-CN.json", import.meta.url);