Graphics

[RossComputerGuy]

Hello, I need help with graphics for my runtime.js operating system. I have been looking at libraries and none of them seems to be what I'm looking for. Things I need to be able to do:

• Load Images
• Load Videos
• Mouse
• Animations
• Shapes
• 2D
• 3D
• Printing Pixels

Edits:

• Could I possibly use node-webcl (a node.js package)?

[facekapow]

Well, runtime is very basic, so this isn't something that has been looked into (I don't think), but you could take a look at some of the terminal code. More specifically the code that controls VGA output. Maybe that'd help.

[facekapow]

About AppJs, no, I don't think so. a) it requires an OS that already has full GUI support, b) it's deprecated, and c) it only supports OSX, Linux, and Windows.

[facekapow]

Node-webcl: I don't think so, it requires the system to have OpenCL support, sorry.

[facekapow]

This got me interested and after a while of reading some OSDev articles on graphics and VGA, I've actually got a decent (initial) implementation of graphics over in the graphics-mode branch, using BGA to initialize graphics mode, and then regular VGA buffers to draw to the screen, using JavaScript to do everything. And it can be used like:

class Range {
  constructor(start, end, inclusive) {
    this.start = start;
    this.end = end;
    this.inclusive = inclusive;
  }
  * [Symbol.iterator]() {
    if (this.inclusive) {
      if (this.start > this.end) {
        for (let i = this.start; i >= this.end; i--) yield i;
      } else {
        for (let i = this.start; i <= this.end; i++) yield i;
      }
    } else {
      if (this.start > this.end) {
        for (let i = this.start; i > this.end; i--) yield i;
      } else {
        for (let i = this.start; i < this.end; i++) yield i;
      }
    }
  }
}

/* setup graphics */
const width = 800;
const height = 600;
const bitDepth = runtime.graphics.constants.VBE_DISPI_BPP_32;
runtime.graphics.enableGraphics(width, height, bitDepth, false, true);
const rawDisplayBuf = runtime.graphics.getDisplayBuffer(width, height);
const displayBuf = new Uint8Array(rawDisplayBuf);

function putPixel(x, y, color) {
  const where = (x * 4) + (y * 3200);
  displayBuf[where] = color & 255;             // blue
  displayBuf[where + 1] = (color >> 8) & 255;  // green
  displayBuf[where + 2] = (color >> 16) & 255; // red
}
/* end graphics setup */

// fill screen width red
for (const i of new Range(0, width)) {
  for (const j of new Range(0, height)) {
    putPixel(i, j, (255 << 16) + (0 << 8) + 0);
  }
}

This will (slowly) fill the screen width red. However, there is a minor bug where it won't fill up the whole screen height, instead it'll only fill about 50 pixels or so in height, if anyone would know how to fix that it'd be great. In the meantime, I'm gonna keep experimenting with it. But it's something! 😄

[facekapow]

I think I fixed it by using an LFB (linear frame buffer) instead of banking (which is restricted to 64kb). The code can now be used like:

/*
 * `width` and `height` can be any size,
 * however I've only tested `bitDepth` with 24 and 32
 */
const width = 800;
const height = 450;
const bitDepth = runtime.graphics.constants.VBE_DISPI_BPP_32;
runtime.graphics.enableGraphics(width, height, bitDepth);
runtime.graphics.getDisplayBuffer().then((rawDisplayBuf) => {
  const displayBuf = new Uint8Array(rawDisplayBuf);

  function putPixel(x, y, color) {
    const where = ((y * width) + x) * (bitDepth / 8);
    displayBuf[where] = color & 255;             // blue
    displayBuf[where + 1] = (color >> 8) & 255;  // green
    displayBuf[where + 2] = (color >> 16) & 255; // red
  }

  // fill screen with white
  displayBuf.fill(0xff);
});

@iefserge What do you think the best approach for graphics would be? The way I see it, there's 3 options:

1. Provide basic graphics setup functions in the kernel library and let the user interact with the raw display buffer (the way I'm currently doing it in graphics-mode)
2. Provide the setup functions in the library, just like option 1, plus convenience methods like putPixel, fillRect, etc. Maybe provide an API similar to the browser's Canvas API.
3. Let external modules provide all graphics APIs.

[iefserge]

This is very cool, graphics can allow some nice things 👍 I think there are a few components:

• VGA driver that works with the hardware and talks to the graphics subsystem
• Graphics subsystem that is abstract from the hardware and allows access to the display. Not sure about raw buffer access, different graphic devices use different pixel formats. It's probably better to provide a library functions (fill, set pixel, shapes etc) and/or virtual buffers (those can be copied into actual video buffer). This subsystem can also load drivers (i.e. VGA driver).
• on top of graphics subsystem there can be a more high level API. I like the idea to use Canvas API here, though this is much harder, it includes complex things like transformation matrices, gradients, text formatting.

I think it's fine to have it all in the same repository, since there is no stable API that can be exposed to modules, nor stable driver API. Having it in one place would make it easier to test and ensure it works in every release.

[piranna]

I think the better aproach is to offer just a low level interface, like Linux framebuffer or also lower, and high level APIs provided by external libraries.

El 3/8/2016 22:21, "Serge" notifications@github.com escribió:

This is very cool, graphics can allow some nice things 👍 I think there are a few components:

• VGA driver that works with the hardware and talks to the graphics subsystem
• Graphics subsystem that is abstract from the hardware and allows access to the display. Not sure about raw buffer access, different graphic devices use different pixel formats. It's probably better to provide a library functions (fill, set pixel, shapes etc) and/or virtual buffers (those can be copied into actual video buffer). This subsystem can also load drivers (i.e. VGA driver).
• on top of graphics subsystem there can be a more high level API. I like the idea to use Canvas API here, though this is much harder, it includes complex things like transformation matrices, gradients, text formatting.

I think it's fine to have it all in the same repository, since there is no stable API that can be exposed to modules, nor stable driver API. Having it in one place would make it easier to test and ensure it works in every release.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/runtimejs/runtime/issues/107#issuecomment-237360100, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgfvm8Ik7UdZPCs6elRxkw0FWADhr28ks5qcPg2gaJpZM4I9KOq .

[iefserge]

@piranna does linux provide access to raw framebuffer or abstraction?

[piranna]

The fbdev interface provide a generic and common interface to access all kind of raw framebuffers. It's mostly a pointer to the graphic card memory with some metadata about color space, line width...

El 3/8/2016 23:22, "Serge" notifications@github.com escribió:

@piranna https://github.com/piranna does linux provide access to raw framebuffer or abstraction?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/runtimejs/runtime/issues/107#issuecomment-237378314, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgfvpzqz5RajZIc8jluoy7iNIl4Yyajks5qcQacgaJpZM4I9KOq .

[facekapow]

@piranna So basically, the Linux framebuffer is the same thing as the framebuffer exposed in runtime.graphics.getDisplayBuffer(), right?

@iefserge If I understood you correctly, the only thing the VGA driver would do would be expose the video memory, right? Everything else like setting up graphics would handled by the graphics subsystem? That's basically what's already in the graphics-mode branch. The only thing would be to add the functions for pixel and shape manipulation instead of providing a raw buffer (I only did it with a raw buffer so that I could test graphics).

[piranna]

@piranna So basically, the Linux framebuffer is the same thing as the framebuffer exposed in runtime.graphics.getDisplayBuffer(), right?

Yes, it seems so.

The only thing would be to add the functions for pixel and shape manipulation instead of providing a raw buffer (I only did it with a raw buffer so that I could test graphics).

I would left that as an independent library/module, so the kernel would be minimal by only allowing access to the low level functions.

[iefserge]

@facekapow yeah, but if it's going to expose a raw buffer, it is going to be specific to hardware (getDisplayBufferVGA?). In this case drawing library can have implementations for each type of graphic buffers.

Just found out that there is also Virtio GPU device in QEMU, which can allow high resolutions and even OpenGL. Haven't tried using it though. But since runtime.js targets mainly hypervisors it might be easiest to support that.

Some links https://lwn.net/Articles/637721/ https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg02966.html http://wiki.qemu.org/ChangeLog/2.5#virtio

[piranna]

@facekapow yeah, but if it's going to expose a raw buffer, it is going to be specific to hardware (getDisplayBufferVGA?). In this case drawing library can have implementations for each type of graphic buffers.

FbDev interface is generic, being VGA, LVMS, bitbanging... It's a generic interface on top of the specific drivers, and also allow hardware acceleration for 2D operations if the driver provides them. Maybe this would be the better approach, offer a low-level naked driver, on top a common interface easy to use for the generic functionality, and later a user space library for high-level functions.

Just found out that there is also Virtio GPU device in QEMU, which can allow high resolutions and even OpenGL. Haven't tried using it though. But since runtime.js targets mainly hypervisors it might be easiest to support that.

That would be nice too, I've read VirtIO drivers are easy to do :-)

Some links https://lwn.net/Articles/637721/ https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg02966.html http://wiki.qemu.org/ChangeLog/2.5#virtio

That looks awesome :-D

[facekapow]

Hmm, how about abstracting away access to the raw buffer and allowing for different graphics handlers? I'm thinking something like the random subsystem, which has different entropy backends. Maybe the same thing could be done for graphics, perhaps with a virtual buffer (Screen) containing Pixels, which store handles to the backend's pixels in order to increase speed (by not having to look up the pixel everytime it's being used).

[iefserge]

That was my initial idea, but looks like Linux simply exposes the raw buffer for better performance (i.e vesafb). Both approaches have pros and cons I guess. It's probably fine to do what's easiest or works best for now, we can refactor it later.

[facekapow]

Ok, I'm running into 2 issues:

1. If I keep the pixels in an array, in order to increase speed, it fails because "there are no more physical pages to allocate",
2. But... if I only get the pixel when it's going to be used, memory usage is very little, however the calculation to find the pixel is run every time, and drawing takes a major speed hit.

[piranna]

Use an ArrayView object to map directly onto the video memory.

El 4/8/2016 18:21, "facekapow" notifications@github.com escribió:

Ok, I'm running into 2 issues:

1. If I keep the pixels in an array, in order to increase speed, it fails because "there are no more physical pages to allocate",
2. But... if I only get the pixel when it's going to be used, memory usage is very little, however the calculation to find the pixel is run every time, and drawing takes a major speed hit.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/runtimejs/runtime/issues/107#issuecomment-237605344, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgfvgXanFeDQgrnz1vV5LaA1jmnQmHWks5qchGagaJpZM4I9KOq .

[facekapow]

@piranna Well, yes, that solves the first issue, and that's the way I've been doing it, but when you do that, then the second issues appears. The point of the virtual buffer is to keep references to the pixels so that you don't have to spend time finding the pixels later. For drawing text or other small things it might be fine to calculate every time, but when using fillRect it slows down significantly.

[piranna]

Create new ArrayViews just for the regions you are usually updating :-) You could also be able to combine an array of ArrayViews to define windows on the framebuffer and use them as mini screens :-D Or also actual UInt8Arrays to implement a double buffer... there's a lot of possibilities... :-D

El 4/8/2016 18:36, "facekapow" notifications@github.com escribió:

@piranna https://github.com/piranna Well, yes, that solves the first issue, and that's the way I've been doing it, but when you do that, then the second issues appears. The point of the virtual buffer is to keep references to the pixels so that you don't have to spend time finding the pixels later. For drawing text or other small things it might be fine to calculate every time, but when using fillRect it slows down significantly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/runtimejs/runtime/issues/107#issuecomment-237609330, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgfvkbCnG5ovVzoxBmkgJORDw5AodtKks5qchT1gaJpZM4I9KOq .

[vitkarpov]

Guys, why do a cloud-based operation system need graphics? :)

[piranna]

People don't consider an OS these days a serious one if the can see high-res pr0n pics on their full HD tv ;-) I didn't wanted that NodeOS had a GUI, but since everybody asked... probably this is due because graphics are more eye-candy than text terminals.

El 4/8/2016 20:55, "Viktor Karpov" notifications@github.com escribió:

Guys, why do a cloud-based operation system need graphics? :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/runtimejs/runtime/issues/107#issuecomment-237648853, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgfvlRPucq7OK_wEjVGQzrK1enO3BDSks5qcjV8gaJpZM4I9KOq .

[vitkarpov]

Seems like simple and lite thing, based on top of the specific idea, turns into general purpose OS as its community grows. Why just not to install Ubuntu? :)

It recalls me evolution of JS MVC framework I used to build: first it has a simple set of features, after a while it already has a lot of stuff, that's not very cool. After that I do really grasp an idea of core and plugins around it :)

[facekapow]

@vitkarpov You're right, that's the reason that I asked if maybe graphics could be included as an external module. While on the thought, I think that the shell should be optional as well, either through an option when requiring runtime.js, or moving it to an external module. Maybe runtime.js should be as modular as possible, because after all, the goal is:

an open-source library operating system (unikernel) for the cloud that runs JavaScript, can be bundled up with an application and deployed as a lightweight and immutable VM image

Basically the only things that are absolutely essential is the ability to run JavaScript, the network, the console, and probably cryptography. If an API for external drivers and services were stabilized, perhaps things like the keyboard and shell, etc. could be moved to external modules.

[piranna]

There's another Javascript unikernel that is diferenciates itself on its readme from both NodeOS and runtime.js because it only expose two functions, require() and syscall(). I think it's not possible to make it more simple :-)

El 4/8/2016 21:26, "facekapow" notifications@github.com escribió:

@vitkarpov https://github.com/vitkarpov You're right, that's the reason that I asked if maybe graphics could be included as an external module https://github.com/runtimejs/runtime/issues/107#issuecomment-237334611. While on the thought, I think that the shell should be optional as well, either through an option when requiring runtime.js, or moving it to an external module. Maybe runtime.js should be as modular as possible, because after all, the goal is:

an open-source library operating system (unikernel) for the cloud that runs JavaScript, can be bundled up with an application and deployed as a lightweight and immutable VM image

Basically the only things that are absolutely essential is the ability to run JavaScript, the network, the console, and probably cryptography. If an API for external drivers and services were stabilized, perhaps things like the keyboard and shell, etc. could be moved to external modules.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/runtimejs/runtime/issues/107#issuecomment-237657410, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgfvoSwXaeCeKYqimXgoEbtTkZEtAXWks5qcjz0gaJpZM4I9KOq .

[iefserge]

I agree that it should be as modular as possible, but what do you think about keeping those things as loadable kernel modules in the same repository? I.e it would be an option in runtime.json

{
  "modules": [
    "virtio-gpu",
    "virtio-rng",
    "vga"
  ],
  "enableGraphics": true
}

The reason for using those kind of modules (instead of external npm modules) is that we don't have to maintain a stable API and can update all the modules with a single PR. We can also read this config in runtime-cli and package only necessary modules into initrd.

(enableGraphics is a separate option because it would most likely need some special support from core.)

[iefserge]

@piranna what unikernel is it? :)

[facekapow]

@piranna I believe it is jskernel, right?

[piranna]

@piranna I believe it is jskernel, right?

Yep :-) I have very bad with the names both of peoples and things, sorry :-P

[facekapow]

@iefserge Having optional modules sounds great, I'm thinking of enableShell and others, too.