Creative developers

[andrevenancio]

I might be biased, but it looks like all my creative coding heroes are now on hic et nunc showcasing their skills from NFT to NFT in a healthy competition to be the first doing this or that.

After some private conversations with some of you, I'm considering writing a proper renderer and trim a bit of the liberties we're currently allowing on SVG's. Not because I don't love what I see, but more of an effort to bring some order to the chaos. I'm thinking of best practices introduced by @mrdoob on his game where he waits for user to click before starting rendering. Though accidental by a game mechanic, it seems like having a way of start and stop rendering on the platform is quite an important thing to have.

I'm imagining scrolling through the feed, and once you see an interactive piece you like, you need to click on it to start rendering the code, and equally stop it once that piece gets off screen or the user clicks a another one.

We can even allow 1 frame to render in order to generate a screenshot so the user has a preview before they click on the piece.

I'm convinced that something with a setup() and a render() method, which has access to a canvas is enough to unleash the creativity we need in a controlled sane way, and maybe once this is available disable javascript on new minted SVG's.

Would love to hear your thoughts and suggestions on what type of reasonable functionalities you expect to have.

Also aware that allowing javascript to be uploaded might require us to sandbox it on an iframe and that might take away some creative tools that otherwise would be available. Like access to webcam for example. So this needs to be a balance between secure and creatively open.

[mrdoob]

I think the main issue are intensive shaders.

One solution would be to provide a template for shaders that takes care of dynamically reducing the resolution of the canvas when not being able to hit 60fps.

That is what <model-viewer> does and that's why having many models on a page do not hang the browser on mobile like those new shaders do.

Another solution that works with svg is to use <img src="foo.svg"> in the stream and <iframe src="foo.svg"> in the objkt page. But I wouldn't know what to do about <iframe>s...

[MAKIO135]

I've been thinking about it a lot lately, and also took a look at the API, I believe it could be good to have a DB to store data when minting an OBJKT, the first frame could be stored as well, and displayed both on the main feed and collections. This might also improve the speed of the site. The on chain data could be pulled only in the OBJKT view. It could also be used to tag abusive/copyminted pieces and prevent their display.

Regarding the renderer, I'm really looking for its development, we must not forget that NFTs can be displayed outside of hicetnunc and should not be too much depend on it to run.

Also, I advice that every parts of the piece, like libraries should be included in the uploaded file in order to be stored on IPFS and not rely on other URLs. They should also always include licenses where needed. This, in order to complete best practices with these new type of pieces 🙂

[jagracar]

I totally agree with makio. I would feel bad to sell a piece that I know it would not be visible in some years because the links are broken or a special keyword (tz wallet) is not passed to the program by the renderer.

[MAKIO135]

disable javascript on new minted SVG's.

SVG animation with JS works just great. The problem might be the abuse of foreignObject. But as long as we have JS running, there will always be ways to hack something either in SVG or canvas, so not sure this is needed.

[andrevenancio]

my problem with the current execution is that there is no control from the user to stop something.

So you scroll down the feed and you have one NFT that generates audio. how do you stop it? refresh the site? what to do if you have 2 nft with audio? you'll be earing both..

I've added an example here: https://twitter.com/andrevenancio/status/1373397478968594432

[andrevenancio]

https://github.com/hicetnunc2000/hicetnunc/tree/feature/new_media and uselocalhost/objkt/9263 to test it

Warning coding standards might burn your eyes

[spite]

the solution for multiple players is to disable them as soon as they leave the viewport, using IntersectionObserver. It can be done by removing them altogether (easy), or disabling all the things that can produce sound (and all the other stuff that we might want to throttle, like requestAnimationFrame, setInterval, etc.) which is way harder.

I think lazy-loading and disabling as things come out of the page is the reasonable option.

The iframes can also auto-implement a wrapper for "Touch to start" and all these features. I have a custom element solution in my website, here's an example: https://www.clicktorelease.com/blog/experiments-with-perlin-noise/. It could implement autoplay, but i chose not to and show a cover image, to respect the user.

[msurguy]

Codepen has published a few tips on how to think about this problem...

https://blog.codepen.io/2014/02/17/can-iframes-annoying-free/ https://blog.codepen.io/2014/04/15/embeds-now-default-run-animations-without-stopping/ https://blog.codepen.io/2020/01/02/web-workers-and-infinite-loops/

[pichiste]

Re: user experience, I really like the idea of providing a more streamlined way of authoring programmable NFTs via the js module template you propose! Maybe it could even have some other nice hooks aside from setup and update that would be handy.

I think @MAKIO135's point about interoperability/longevity is an important one though. One solution could be that the user uploads just the js file, but upon preview/upload it gets automatically bundled (inline) into an html document, along with any libs or assets. What ends up on IPFS is just the standalone HTML file that will work anywhere, but the user never has to deal with the HTML directly. You'd then display it in an iframe as it's being done now. I've been working on a project boilerplate that does exactly this kind of inline bundling via rollup and would be happy to share if it's useful.

Re: the start/stop issue, @spite's suggestion makes a lot of sense to me. The cover image could even be embedded in the HTML file and extracted on load so that it's possible to toggle between cover and iframe via intersectionObserver. Maybe the cover image issue is something that needs addressing across the different media types though?

One last thing about iframes: It would be a shame to not have the ability to access features like cam, webxr, fullscreen, etc... to me that blocks a huge amount of potential. I wonder if it's more about making a carefully selected feature policy to apply to the sandboxed iframes?

[mrdoob]

Ideas proposed on the discord chat:

HTML

1. Ask the user to provide a thumbnail.
2. Convert (and resize if necessary) to base64.
3. Inject it in <head> as <meta property="og:image" content="data:image/png;base64,..." />.
4. In the feed use <img> to show the thumbnail.
5. In the objkt page use <iframe>.

SVG

1. In the feed use <img src="foo.svg">.
2. In the objkt page use <iframe src="foo.svg">.

[MAKIO135]

Could we switch between the 2 types of renderers (static/dynamic) on mouseenter/mouseleave/tap on main feed and collections? This would make it easier to get a preview of the piece

[DonKarlssonSan]

I agree on taking inspiration from CodePen as @msurguy suggests. They have the exact same problem: displaying a lot of previews of stuff that risk hogging the computer and annoy the user.

It's a lot to think about:

• Sound - mute by default, maybe can't trust every creator to start sound after user clicks
• Performance - SVG animations, CSS animations, canvas animations, loops, computations, setInterval()
• User privacy - webcam, mic - the user has to accept in the browser but still
• Creator flexibility - if it's possible, I'd like to be able to use all and every browser feature =)
• Longevity - agree with @MAKIO135 that NFT's should be able to run independent/outside of hicetnunc

[DonKarlssonSan]

The sandbox provides security inside the browser, but is there a need to worry about what the executing code does? Like mining bitcoins and such.

[mrdoob]

How can it mine bitcoin if it can't do external connections?

[ricardomatias]

I think for dynamic NFTs (nothing rendered), they should have a label/badge/icon to show that they are such and on hover load/ run them. A screenshot isn't a good medium to transcribe interactivity or show development and wouldn't work with pieces which have audio.

[andrevenancio]

I think for dynamic NFTs (nothing rendered), they should have a label/badge/icon to show that they are such and on hover load/ run them. A screenshot isn't a good medium to transcribe interactivity or show development and wouldn't work with pieces which have audio.

A screenshot isn't a good medium but its better than having no preview at all. If you think of the NFT in the context of the feed, to the user its just empty square with a "Play" button. Having an image will definetely help you visually filter what you want to see. IMHO...

[ricardomatias]

If it's the same for everyone, you could argue it creates a disadvantage towards still pieces. On the other hand, is the purpose of the feed to make it efficient for the user to scan artworks? I think it'd be cooler to create engagement rather than follow social network practices since the intent is different (in my perspective). Abstract things aside, we still need a way to "thumbnail" audio artworks.

[mrdoob]

The intent is to not hang people's devices while browsing the feed because one of the interactive pieces have a intensive shader.

You can even crash the whole tab and make the site inaccessible for users with slow gpus.

[ricardomatias]

@mrdoob that's a bit off, I didn't suggest for just running them on load.

[pichiste]

the solution for multiple players is to disable them as soon as they leave the viewport, using IntersectionObserver. It can be done by removing them altogether (easy), or disabling all the things that can produce sound (and all the other stuff that we might want to throttle, like requestAnimationFrame, setInterval, etc.) which is way harder.

I think lazy-loading and disabling as things come out of the page is the reasonable option.

The iframes can also auto-implement a wrapper for "Touch to start" and all these features. I have a custom element solution in my website, here's an example: https://www.clicktorelease.com/blog/experiments-with-perlin-noise/. It could implement autoplay, but i chose not to and show a cover image, to respect the user.

Something along these lines makes the most sense to me. There should probably never be more than 1 iframe running at any given time.

[andrevenancio]

I think this issue can now be closed. Thanks for all the nice feedback everyone!