Create `gemoji` type to support compiling back to shortcodes

[kellyjosephprice]

Initial checklist

• [X] I read the support docs
• [X] I read the contributing guide
• [X] I agree to follow the code of conduct
• [X] I searched issues and couldn’t find anything (or linked relevant results below)

Problem

I'd like to be able to render gemoji shortcodes as unicode emojis in html/react, but I'd like to be able to compile them back to shortcodes.

Take the original example:

Look, the moon :new_moon_with_face:

Here’s a family :family_man_man_boy_boy:

Слава Україні!  :ukraine:

import remarkGemoji from 'remark-gemoji'
import remarkParse from 'remark-parse'
import remarkStringify from 'remark-stringify'
import {read} from 'to-vfile'
import {unified} from 'unified'

const file = await unified()
  .use(remarkParse)
  .use(remarkGemoji)
  .use(remarkStringify)
  .process(await read('example.md'))

console.log(String(file))

I'd like the output to be the original doc:

Look, the moon :new_moon_with_face:

Here’s a family :family_man_man_boy_boy:

Слава Україні!  :ukraine:

Solution

I'd like to propose adding an option to transform the shortcodes into nodes?

import remarkGemoji from 'remark-gemoji'
import remarkParse from 'remark-parse'
import {read} from 'to-vfile'
import {unified} from 'unified'

const processor = unified()
  .use(remarkParse)
  .use(remarkGemoji)
const tree = await processor.run(
  processor.parse(':wave:')
)

console.log(String(tree))

{                                                                                           
  children: [                                                                               
    {                                      
      children: [                          
        {        
          name: 'wave',
          type: 'emoji',   
          value: '👋'
        }
      ],                                   
      position: {
        end: {                                                                              
          column: 7,
          line: 1,
          offset: 6
        },
        start: {
          column: 1,
          line: 1,
          offset: 0
        }
      },  
      type: 'paragraph'
    }                                                                                       
  ],                                       
  position: {                                                                               
    end: {     
      column: 7,
      line: 1,
      offset: 6
    },
    start: {     
      column: 1,  
      line: 1, 
      offset: 0
    }
  },       
  type: 'root'
} 

Alternatives

I don't think so? By transforming it to text, it's impossible to tell if it was an gemoji or unicode.

[wooorm]

Hey Kelly!

I'd like the output to be the original doc:

A fundamental property of ASTs is that it’s impossible to generate the original doc from them. ASTs intentionally do not contain everything. So, you can never get the output be the input.

---

Could you expand on the problem you’re running into?

[kellyjosephprice]

Could you expand on the problem you’re running into?

I'm working on a markdown backed, semi-WYSIWYG editor. On save the editor writes back out to markdown and it tries to preserve as much off the original doc as possible. So I'd like to be able to detect if it was originally written as unicode or shortcodes, and write it out the same way.

Thinking about it again, the positional data would be enough for my use case. And I already have a functioning transformer. But, I got half way through writing up a PR but figured I should ask if you'd be receptive first.

Thanks!

[wooorm]

Right, as I mentioned before: you will run into 1000s of cases where that isn’t possible because of ASTs. Solving this one, will still leave 999 cases unsolved.

It all depends a lot on what you’re doing exactly, but wouldn’t it be better to:

• let people write whatever in markdown
• in the semi-WYSIWYG stuff, use perhaps the AST, but also https://github.com/Rich-Harris/magic-string to make tiny edits on the markdown source
• separately, use this project (and others) to render things to HTML (without saving back to markdown)

[kellyjosephprice]

Solving this one, will still leave 999 cases unsolved.

Certainly, I'm not trying to solve them all. And it's definitely not critical.

• in the semi-WYSIWYG stuff, use perhaps the AST, but also https://github.com/Rich-Harris/magic-string to make tiny edits on the markdown source

Oh that's an interesting approach.

separately, use this project (and others) to render things to HTML

We do and have been for some time! Thanks for all your work!

[wooorm]

Thinking about it again, the positional data would be enough for my use case. And I already have a functioning transformer. But, I got half way through writing up a PR but figured I should ask if you'd be receptive first.

I hope you don’t mind but I’d prefer not to do this here. Custom nodes for things is something we do for syntax extensions, and this isn’t really that. The AST is simple by dropping all that “useless” info. Which makes plugins easier. That’s the ideology here. Making those trees more complex again, defeats the purpose a bit.

It’s perhaps the best way to go about your problem, but I’m not convinced I want to maintain that code!

All the best :)

[github-actions[bot]]

Hi! This was closed. Team: If this was fixed, please add phase/solved. Otherwise, please add one of the no/* labels.