Graphics/Geom3D discussion at JuliaCon?

[protogeezer]

If others are interested, I'd like to meet at the conference to discuss Gadfly backends or drivers, mostly, and seek advice on how to approach contributing to a broader Julia graphics architecture.

My main interest is bringing "Geom3D" visualization to Gadfly by exploiting the VTK 3d C++ graphics classes. A secondary goal is to broaden the backend to be more media oriented - which can be thought of as making features like Reactive intrinsic.

There are smaller steps that might also benefit the community. Here is a brief synopsis of what I'm interested in and where I got to before hitting one kind of obstacle or another: (1) A window-based alternative to the browser/Snap display which would be much easier if the REPL didn't hang the main process (or notifier) thread. It doesn't make much sense to try to fix that in Gadfly/Compose. (This would retire the timer hack to work around the hung notifier). So, how does one reach consensus on what a solution would entail that would be acceptable to the broader Julia project? (2) native Cocoa or Qt drivers for Compose? Native drivers would make it easier to add LuaTex derived labels - as pdf fragments, non-LtR text, and pub quality 2D image output - a side-effect of the native support for displaying PDF fragments. I suspect that it would be a lot faster. (3) in particular, (2) depends on OTF fonts. Dropping FreeType for Cocoa and/or Qt text functionality makes it much easier to accomplish first class text layout, in general. (4) Adding basic features to the Gnome/GTK Julia base depends on building custom, or at least newer, versions of at least Cairo, Poppler, and the Gnome or FreeType text layout libraries than are available in Linux or the OSX HomeBrew capability. What is the process for proposing updates to the Brew distributions? How would this work in Linux? (5) Gnome/GTK Compose backend would need it's own (PoDoFo or Poppler based?) pdf parser and layout engine - assuming LuaTex, e.g.. Also, most of the new features mentioned above would use the Cairo recording surface - or at least my prototype implementation of a window based display uses it. The current version of Cairo has a bug in the recording surface that prevents it from working with text layouts using certain coordinates (see Cairo bug #89232). For whatever reason, this didn't manifest itself until I added OTF fonts to the prototype. It's indeterminate when the bug fix will make it into the main Cairo distribution. For now it seems to be relegated to the December Cairo snapshot 1.15.4. (6) Geom3D implementations based on VTK would require Qt and/or Cocoa drivers. Currently, the GTK support in VTK is limited to the Python wrappers. I seems that a desirable outcome would be for Julia to supersede Python not become more dependent on it?

It seems that Julia is, possibly by fiat, supporting GTK. Before undertaking a serious effort to contribute the features listed above to Gadfly/Compose, it would be reassuring to reach some kind of consensus with both the Gadfly community and the broader Julia project.

[lobingera]

I have to say i cannot fully capture what you are proposing to do, planning to do and trying organizing to do. And btw: for a better audience i recommend a discourse posting.

Could it be your mixing up FreeType and Pango? FreeType is a font rendering library and more or less the industry standard but is not doing any layout.

[bjarthur]

3D plots and more interactivity would be great!

would GLVisualize.jl work as an alternative to VTK? i believe @simondanisch is working on a pure julia version of it.

is there a way to make 3D work across all current backends?

re. interactivity-- i wonder how much more we could squeeze out of the javascript embedded in SVG.

whatever you end up doing, please break it up into small independent chunks. much easier to review PRs that way.

and thanks for starting a discussion to reach a consensus. many folks don't and then end up forking when their work doesn't get merged.

[protogeezer]

@lobingera - what is a discourse posting?

My intention is to come to JuliaCon to discuss this directly with as many core developers as possible to learn how to fit into the overall project?

The goal of contribution is to use VTK to implement a succession of 3D Geoms - initially as a means for comparing performance and bang-for-the-buck of using an industry standard C++ library. Is VTK to graphics what LLVM is to the core Julia functionality?

What I was trying to (overly briefly) describe some of the technical steps along the way - as well as some of the obstacles with the Julia architecture. The lack of support for acquiring and releasing window focus in the REPL is one example. Where to identify solutions to such issues?

If it would be a simpler to do the first implementation of the 3D Geoms with macOS/Cocoa, would that be acceptable? Qt would be harder to work with, so can that middleware layer wait for a "production implementation?" Finally, if Gnome/GTK is just not an option, is that a problem?

@bjarthur - good I'm glad that 3D plus more interactivity would be a good thing. GLVisualize is an OpenGL (or GLUT) backend, or renderer, that can be used with Gadfly and some other plotting packages. See the GLVisualize examples.

VTK is a gigantic set of C++ classes (see http://www.vtk.org/doc/nightly/html/classes.html) that include renderers, but more importantly, collections of classes for reading and writing graphics and mesh oriented data formats, defining "plots" in terms of the VTK class hierarchy, and filters for processing data streams on the fly. It is probably the right idea to think of VTK of a superset of what can be done with Compose + the Gadfly backends. This is not too say that Compose will be completely displaced.

One advantage of the VTK approach is that it is possible to update the object hierarchy for a plot with new data, for example, and maybe update other objects such as labels, then fire off a render - an efficient scheme for realtime animation assuming it can be handled in a pipeline of some sort.

Ben hits the nail on the head at the end of his post. I don't want to initiate a large body of work that doesn't end up being accepted into the larger Julia project. I need advice on how to achieve the consensus about how best to proceed.

I can see why it's important to break up the project into components. What is less clear is how to handle the complexity of adding components to either the OS X home-brew scheme or packages for the linux platforms.

[lobingera]

@protogeezer: https://discourse.julialang.org/

[SimonDanisch]

Iif it helps: I'm currently working on https://github.com/SimonDanisch/Visualize.jl which will decouple the declaration of drawing from the actual backend. It's pure Julia and I compile the Julia shaders to OpenGL for the OpenGL backend, to have a performant backend. But - since it's all Julia - it's also possible to take parts of the pipeline, and emit e.g. cairo commands at the appropriate level, to e.g. generate PDFs with 3D content described as 2D vector graphics. I hope that i'll be able to show some cool demos at juliacon :)

[SimonDanisch]

One advantage of the VTK approach is that it is possible to update the object hierarchy for a plot with new data, for example, and maybe update other objects such as labels, then fire off a render - an efficient scheme for realtime animation assuming it can be handled in a pipeline of some sort.

That has been the model in GLVisualize so far as well. In Visualize I aim for a much more flexible trait system, which would allow painting like in Cairo/Compose but also conserve the GLVisualize/VTK model, with as little API and code churn as possible. Let's see how that works out :D

[protogeezer]

@SimonDanisch Seeing the demos would be helpful - or at least it would provide a point of departure for a discussion about two divergent approaches for implementing a graphics/visualization infrastructure.

I didn't see a session where you're speaking so perhaps we can do introductions after the workshop on Tuesday.

I believe that the distinction between the two ideas is that what you are proposing is logically an upgrade to Compose while I'm thinking of taking the implementation of Geoms completely out of julia by using the class model provided by VTK? We'll see. It's worth pursuing both until the strengths and weaknesses of both paths become apparent.

[SimonDanisch]

This is my talk: http://juliacon.org/2017/talks#talk-8

implementation of Geoms completely out of julia by using the class model provided by VTK

Well, the advantage is obviously that you can rely on something very solid and on vaste functionality. On the other hand, VTK has a lot of technical debt and is really hard to extend. Also, I think some features will be incredibly hard to implement in C++ & VTK - which is where a pure julia version will shine

[protogeezer]

Well you got me there. I wasn't actually planning on spending Thursday at JuliaCon. I'll come see the talk.

I am interested in understanding your point of view. The issue that colleagues and others identified when they/we tried to use Julia is that the base language is very expressive (by that I mean it concisely captures intent). Our collective experience is that there is a wall when other kinds of tasks are attempted - such as, not surprisingly - visualization.

I really want to understand where you see julia shining on things that would be hard to implement in C++... I haven't observed examples of those things yet. There is also the issue of performance - an example that largely paralleled my own experience is Mocha.jl - see "Portability and Speed." That is also a model for where I could eventually take the VTK contribution.

Something that blends well might be beneficial?

[SimonDanisch]

Mocha.jl - see "Portability and Speed."

Well, not sure how telling that is. I mean it's using OpenMP, which is no secret that the pure julia alternatives aren't great yet! So 2x is actually pretty good in that context :) I haven't heard of many graphics libraries using OpenMP, though ;)

julia shining on things that would be hard to implement in C++.

Jeez, there are so many things... E.g. Transpiler && GPUArrays

I can write Julia functions that run on the GPU, how awesome is that? I can write a graphics program in Julia, run it via OpenGL, and accept user callbacks that will end up running in the OpenGL shader...

And I created these packages as a relatively small side project... I can in no way imagine how to do this in C++ in a timely manner. And I believe that the resulting code is pretty user extensible and easy to grow, considering the performance && flexibility it yields

Also have a look at stuff like https://github.com/JuliaDiff/ReverseDiff.jl

But in the end, the clarity of the language already does big favors for me - I can do things in much less time...

Our collective experience is that there is a wall when other kinds of tasks are attempted

Can you elaborate on this? Or do you simply mean the lack of packages readily available? The latter is definitely true, but I see no reason why this couldn't change.

[protogeezer]

re: Our collective experience is that there is a wall when other kinds of tasks are attempted

Can you elaborate on this? Or do you simply mean the lack of packages readily available? The latter is definitely true, but I see no reason why this couldn't change.

Off the top of my head the sorts of things that have scuttled previous attempts of enhancements for my own work - The build system regularly fails during updates - it doesn’t work (on macOS) with -p invoked. The REPL hangs the main process thread (I have no idea where to discuss or how to propose fixes to enhance the REPL to pay attention to acquire/loss of focus events). Fixing that would make it possible to have GUI interfaces for visualization that don’t depend on the timer hack. CMake for the build system would make it easier to configure and build more complex configurations - including parallel builds - both during development and deployment - for example configure REPL for Cocoa event processing and build Cairo using Cocoa, etc, etc. It would be much easier to do true cross platform builds if both Linux, macOS, etc used CMake - no home-brew on macOS… Subbing out Gnome/GTK for Qt and or Cocoa would make it possible to be more cross platform - rather than essentially porting Gnome to the other platforms as an LCD. Qt on windows and macOS is more of an abstraction layer - much closer to the native OS than GTK/Gnome/Xwindows, etc.

A couple of times a week julia crashes with Sig 11 or some such thing.

On Jun 14, 2017, at 4:20 PM, Simon notifications@github.com wrote:

Mocha.jl - see "Portability and Speed."

Well, not sure how telling that is. I mean it's using OpenMP, which is no secret that the pure julia alternatives aren't great yet! So 2x is actually pretty good in that context :) I haven't heard of many graphics libraries using OpenMP, though ;)

julia shining on things that would be hard to implement in C++.

Jeez, there are so many things... E.g. Transpiler && [GPUArrays](https://github.com/JuliaGPU/GPUArrays.jl]

I can write Julia functions that run on the GPU, how awesome is that? I can write a graphics program in Julia, run it via OpenGL, and accept user callbacks that will end up running in the OpenGL shader...

And I created these packages as a relatively small side project... I can in no way imagine how to do this in C++ in a timely manner. And I believe that the resulting code is pretty user extensible and easy to grow, considering the performance && flexibility it yields

Also have a look at stuff like https://github.com/JuliaDiff/ReverseDiff.jl

But in the end, the clarity of the language already does big favors for me - I can do things in much less time...

Our collective experience is that there is a wall when other kinds of tasks are attempted

Can you elaborate on this? Or do you simply mean the lack of packages readily available? The latter is definitely true, but I see no reason why this couldn't change.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[SimonDanisch]

Okay, that seems completely orthogonal...

[lobingera]

@protogeezer Which build system regularly fails? What do you mean with "build Cairo using Cocoa"? And how do you come to the conclusion, that "It would be much easier to do true cross platform builds if both Linux, macOS, etc used CMake - no home-brew on macOS"? How would you 'use' Cocoa on Win (esp. Win10)?

[protogeezer]

@SimonDanisch - no, in my opinion, starting a sizable enhancement effort without a clear means to deploy the contribution seems pointless. The same is true wrt the REPL/Runtime not being window manager aware.

@lobingera Which build system regularly fails? It turns out that the "main" julia build system doesn't work if a beta release of macOS is on the machine. See below for the hiccup after reinstalling 10.12.5. So it is still unclear if I have a useful build or not.

What do you mean with "build Cairo using Cocoa"? Sorry about that. I mean that I want/need to be able to build Cairo with the quartz font, surface and image backends enabled AND the X components explicitly disabled. And probably the recording surface - unless that can be subsumed into the VTK GEOMs directly.

And how do you come to the conclusion, that "It would be much easier to do true cross platform builds if both Linux, macOS, etc used CMake - no home-brew on macOS"? the simple answer is that I have contributed CMake scripts to several other open source (scientific) packages.

How would you 'use' Cocoa on Win (esp. Win10)? I wouldn't. I would use the cairo windows surface and font modules. Since I don't use windows I don't know if the win32 backends are suported on windows 10 or not.

The other alternative is to use the cairo Qt surface, which I believe can be built to use the Qt windows (10) or cocoa widget/window. I don't like that as much as doing a "pure" cocoa prototype or testbed to see what is feasible in terms of speed and quality of rendering.

The difference between Qt and Gnome is that Qt uses, and has a means to (directly) expose the underlying OS. I haven't found examples of anyone doing that with Gnome.

==================

et-imac-retina:~ sjbespa$ julia-0.6 () | A fresh approach to technical computing () | () () | Documentation: https://docs.julialang.org | |_ | Type "?help" for help. | | | | | | |/ ` | | | | || | | | (| | | Version 0.6.0-rc3.0 (2017-06-07 11:53 UTC) / |_'|||_'_| |
|/ | x86_64-apple-darwin16.6.0

julia> Pkg.add("Cairo") INFO: Updating cache of Homebrew... INFO: Installing Cairo v0.3.0 INFO: Installing Graphics v0.2.0 INFO: Installing Homebrew v0.5.7 INFO: Building Homebrew Cloning into '/Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Library/Taps/homebrew/homebrew-core'... remote: Counting objects: 4439, done. remote: Compressing objects: 100% (4243/4243), done. remote: Total 4439 (delta 35), reused 465 (delta 12), pack-reused 0 Receiving objects: 100% (4439/4439), 3.53 MiB | 4.87 MiB/s, done. Resolving deltas: 100% (35/35), done. remote: Counting objects: 353895, done. remote: Compressing objects: 100% (145932/145932), done. remote: Total 353895 (delta 209006), reused 350037 (delta 205159), pack-reused 0 Receiving objects: 100% (353895/353895), 113.09 MiB | 10.71 MiB/s, done. Resolving deltas: 100% (209006/209006), completed with 3647 local objects. Cloning into '/Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Library/Taps/staticfloat/homebrew-juliadeps'... remote: Counting objects: 1868, done. remote: Total 1868 (delta 0), reused 0 (delta 0), pack-reused 1868 Receiving objects: 100% (1868/1868), 354.37 KiB | 0 bytes/s, done. Resolving deltas: 100% (1118/1118), done. Cloning into '/Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Library/Taps/staticfloat/homebrew-juliatranslated'... remote: Counting objects: 5, done. remote: Total 5 (delta 0), reused 0 (delta 0), pack-reused 5 Unpacking objects: 100% (5/5), done. Reset branch 'stable' error: tag '9.9.9' not found. remote: Counting objects: 12, done. remote: Compressing objects: 100% (8/8), done. remote: Total 12 (delta 5), reused 11 (delta 4), pack-reused 0 Unpacking objects: 100% (12/12), done. Note: checking out '9.9.9'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:

git checkout -b

HEAD is now at b38c52f93... Merge pull request #2289 from reitermarkus/cask-refactor-uninstall INFO: Building Cairo WARNING: BinDeps.shlib_ext is deprecated. likely near /Users/sjbespa/.julia/v0.6/Cairo/deps/build.jl:112 Warning: Already linked: /Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Cellar/pcre/8.40 Warning: Already linked: /Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Cellar/glib/2.52.2 Warning: Already linked: /Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Cellar/pixman/0.34.0 Warning: Already linked: /Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Cellar/libpng/1.6.29 Warning: Already linked: /Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Cellar/freetype/2.8 Warning: Already linked: /Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Cellar/fontconfig/2.12.1_2 Warning: Already linked: /Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Cellar/cairo/1.14.8 Warning: Already linked: /Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Cellar/gdbm/1.13 Warning: Already linked: /Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/Cellar/pkg-config/0.29.2 Warning: The post-install step did not complete successfully ==============================================================[ ERROR: Cairo ]==============================================================

LoadError: failed process: Process(/Users/sjbespa/.julia/v0.6/Homebrew/deps/usr/bin/brew install --ignore-dependencies staticfloat/juliatranslated/openssl, ProcessExited(1)) [1] while loading /Users/sjbespa/.julia/v0.6/Cairo/deps/build.jl, in expression starting on line 162

============================================================================================================================================

==============================================================[ BUILD ERRORS ]==============================================================

WARNING: Cairo had build errors.

• packages with build errors remain installed in /Users/sjbespa/.julia/v0.6
• build the package(s) and all dependencies with Pkg.build("Cairo")
• build a single package by running its deps/build.jl script

============================================================================================================================================ INFO: Package database updated INFO: METADATA is out-of-date — you may not have the latest version of Cairo INFO: Use Pkg.update() to get the latest versions of your packages

julia> Pkg.update() INFO: Updating METADATA... INFO: Updating cache of Media... INFO: Updating Gadfly master... INFO: Computing changes... INFO: Upgrading Media: v0.2.7 => v0.3.0

[protogeezer]

@lobingera where do I go from here? I'm frustrated because I feel like I'm trying to carry on a conversation using a couple tin cans and a handful of straws. I'm barred from posting to discourse because I'm not "trust level 2." So, to return to the original purpose of the post - finding people that can help identify a development plan that would result in a more comprehensive visualization layer than is currently possible - is it even worth coming to JuliaCon? Who should I be trying to connect with if it is the right venue to pursue my goals?

[lobingera]

@protogeezer I don't know, i just checked with the discourse FAQ, actually already a TL=0 you should be able to post (but not everything, you're limited with attachments and links) if the admin has not decided differently. Afaics i'm TL=1 and can post...

In any case if you have the opportunity to go to such a meeting (i haven't) do it.

For the work/project overall, as you can see from my above comments, it's like i cannot follow a story line here, what components you see necessary, what you are working on right now etc. but that might be clearer in a F2F meeting.

[timholy]

I definitely recommend coming to JuliaCon and watching @SimonDanisch's presentation, which I'm sure will be amazing. That might set a good foundation for productive discussion.

[protogeezer]

Tim - Yes, I’ll come to JuliaCon to learn about Julia at a much deeper level than I’ve garnered on my own. I do hope to have constructive discussions too.

What’s already there in GLVisualize is something to study carefully and emulate. However, I think that the comparison between VTK and LLVM is apt. I’m not sure there is time to reproduce the complex algorithms in VTK in Julia in any reasonable amount of time. Admittedly, there are different approaches that all have merit, and I hope to leave JuliaCon with a better understanding on how to contribute something significant.

Stephen

On Jun 16, 2017, at 2:50 PM, Tim Holy notifications@github.com wrote:

I definitely recommend coming to JuliaCon and watching @SimonDanisch's presentation, which I'm sure will be amazing. That might set a good foundation for productive discussion.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[SimonDanisch]

@protogeezer sorry, I didn't want to demotivate anything. It just seems like you're interested in an orthogonal set of problems - not that it is any less relevant!

[protogeezer]

Fair enough. See you next week. I look forward to an interesting meeting.

On Jun 16, 2017, at 7:03 PM, Simon notifications@github.com wrote:

@protogeezer sorry, I didn't want to demotivate anything. It just seems like you're interested in an orthogonal set of problems - not that it is any less relevant!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[lobingera]

@protogeezer I asked about your posting problem on discourse https://discourse.julialang.org/t/barred-from-posting-to-discourse-because-im-not-trust-level-2/4309

[lobingera]

@protogeezer Can you report something about your meeting?

[protogeezer]

Andreas:

Sure. I’m busy for the next couple of days. I’ll write up something mid-week. (It was worth the trip!)

Stephen

On Jul 2, 2017, at 2:03 AM, Andreas Lobinger notifications@github.com wrote:

@protogeezer Can you report something about your meeting?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[protogeezer]

I’m not sure what I expected from the conference, but in the end it was a satisfying experience. After conversations with other users and a few of the Julia Computing developers I think I have a better understanding of how to approach creating a variant(?) of Julia that is window manager aware.

I learned why Julia was implemented the way it is. Simply put, Julia has to run on machines that may not even have window managers installed. So, it’s not going to be a one-way-or-the-other proposition vis-a-vis a window manager. Both types of “use cases” have to be accounted for. This was apparent with the nearly mind-boggling breath of topics that were covered in the talks - and a nearly equal number of “use cases” - web-based, notebook UI, GUI apps, and pure REPL based projects. There is definitely not going to be a one-size-fits-all approach.

The gist of the technical approach is to, initially at lease, implement the libUV interface using either Cocoa, or Qt, and ultimately both. When I finally submit a patch, it will be viewed in a more favorable light if it includes a Make.user switch, with a default of value of “off," that will enable the user to build a “window manager” version of Julia or the current (for lack of a better term) "bare bones Unix” implementation. There was definitely not an NFW attitude in the least - in fact there was a fair amount of encouragement to give it a shot.

The talks most directly pertinent to Julia graphics were on GLVisualize and the Julia implementation of the GR Framework. A talk that may serve as a harbinger was on the new developments in the Julia implementation of QML. It was quite impressive. QML also showed up in the GR talk. QML as the Julia GUI?

It was also invigorating to get in touch with my inner-nerd by talking to users that had opinions about such topics as “type stability,” and “Julia as a systems programming language.”

I wasn’t able to stay for the Hackathon. Maybe next time.

Stephen

On Jul 2, 2017, at 2:03 AM, Andreas Lobinger notifications@github.com wrote:

@protogeezer Can you report something about your meeting?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ihnorton]

implement the libUV interface using either Cocoa, or Qt, and ultimately both.

@protogeezer I would suggest to make very, very sure you have buy-in from some of the core developers who would actually need to approve this, before spending any considerable time on it. I fear you may have gotten the wrong impression as there's very close to zero chance a Qt dependency -- even an optional one -- would be accepted in base. The reason is that it is completely unnecessary to introduce that level of coupling. Such things can be fully realized as packages, as GLVisualize, QML, GTK, and several other projects demonstrate.

That's not to discourage windowing-related development (far from it, such efforts are great! 💯) but to make sure any substantial efforts are not misplaced in a direction that won't be integrable in base.

It is true that there are a number of issues complicating interactivity (multithreading safety and re-entrancy, GC and compilation pauses, etc.), but they have very little to do with libuv per se, and wouldn't be mitigated by replacing the event loop or i/o system.