Can't get labels and lines from an indicator

[MakiBM]

I'm using this indicator

    indicators: [
      { name: 'Divergence for Many v4', id: 'PUB;5xi4DbWeuIQrU0Fx6ZKiI2odDvIW9q2j', version: '10' },
    ]

and the script doesn't read anything from it

I've been debugging that a bit and first of all this is seems to be due to usage of line to produce the drawings on the screen.

I found it to sit in socket messages with dwglines / dwglabels keys on it

// Puppeteer
client.on('Network.webSocketFrameReceived', ({response}) => {
    console.log(
      response.payloadData.split(/~m~\d*~m~/)
        .filter(str => str.includes('dwglabels'))
        .map(JSON.parse)
        .map(msg => JSON.parse(msg.p[1].st4.ns.d))
        .map(d => d.graphicsCmds.create.dwglabels[0].data)
        .flat()
    )
})

Was hoping to get you a PR that would cover that corner case, however I went into rabbit hole with the indexing. Can't wrap my head around how to map it back to bars / timestamps.

Is it something you're aware of? I guess I could contribute but would need some guidance on indexing. I didn't spot anything special about that in the rest of the script.

Here is data structure from that object

[Mathieu2301]

Wow okay I see the problem ! First, if you want to contribute, please look at the v3 branch, I have redone all the library and it is much better organized. I'll take a look now...

[Mathieu2301]

I manage to implement it but I don't know what to name the new data. I don't know if we can find a pattern that would work with all the indicators. I think dwgLines appears when line.new( ... ) is called in the Pine script. Same for dwgLabels and label.new( ... )

[Mathieu2301]

Do you know what 'ns' could means in data.ns ?

[Mathieu2301]

In the PineScript documentation we can see all (maybe) the extra data types we have to parse.

So we have know how to parse :

• Labels
• Lines
• Boxes
• Tables

To do it, maybe we can create a study that uses all these types of data.

[Mathieu2301]

The parsed json data.ns.d can contain :

• data.report.performance If the indicator is a strategy. We put it in this.#strategyReport.performance.
• dataCompressed If the indicator is a strategy. We parse it and put in this.#strategyReport.
• graphicsCmds If the indicator is drawing Labels, Lines, Boxes or Tables.
• Maybe something else ? It would be interesting to know.

So I think we just have to parse graphicsCmds (using a custom pine study). We can imagine this type of use :

const TradingView = require('../main');

const client = new TradingView.Client();

const chart = new client.Session.Chart();
chart.setMarket('BINANCE:BTCEUR', {
  timeframe: 'D',
});

TradingView.getIndicator('PUB;5xi4DbWeuIQrU0Fx6ZKiI2odDvIW9q2j').then((indic) => {
  const DFMI = new chart.Study(indic);

  DFMI.onUpdate((changes) => {
    if (!changes.includes('graphic')) return;
    console.log('Labels =>', DFMI.graphic.labels);
    console.log('Lines =>', DFMI.graphic.lines);
  });
});

And DFMI.graphic.labels would look like :

[
  {
      id: 5, // Label ID
      x: 2, // X position (Should be converted to timestamp ? how ?)
      y: 8669.55, // Y position (Price)
      yl: 'pr', // Maybe 'yLock' ? ('pr' = 'price')
      t: 'RSI\nStoch\nOBV\nMFI\n4', // Label title/content
      st: 'ldn', // Maybe label position ('ldn' or 'lup' (or ??))
      ci: 1, // I dont know what it is (color ?)
      tci: 5, // I dont know what it is
      sz: 'normal', // Maybe 'size' ?
      ta: 'center', // Maybe 'text align' ?
      tt: '', // Maybe 'tooltip' ?
  },
  ...
]

DFMI.graphic.lines would look like :

[
  {
      id: 5, // Line ID
      x1: 0, // X1 Position (Should be converted to timestamp ? how ?)
      y1: 8050, // Y1 Position (Price)
      x2: 1, // X2 Position (Should be converted to timestamp ? how ?)
      y2: 8598, // Y2 Position (Price)
      ex: 'n', // Maybe 'extend' ? extend = ('none', 'both', 'right' or 'left'), and 'n' = 'none'
      st: 'sol', // Maybe 'style' ? style = ('solid', 'dotted', 'dashed', 'arrow_left', 'arrow_right' or 'arrow_both'), and 'sol' = 'solid'
      ci: 1, // Maybe 'color' ?
      w: 2, // Maybe 'width' ?
  },
  ...
]

Btw, I think graphic.labels, graphic.lines, etc... must be ordered by date (first is oldest), unlike for plots (first is the most recent).

I also want to rename all keys :

• yl => yLock
• t => title or content
• st => position
• etc...

And same for values :

• 'n' = 'none'
• 'sol' = 'solid'
• etc...

Is it OK for you ? If you want to try something, go in src\chart\study.js after line 205 :

I hope that with all this we can go in the same direction 😁 And if you have an idea of what data.ns.indexes is, please tell me 👌

[Mathieu2301]

I saw interesting things on pine-facade.

I put this here, maybe it can help.

And in my last message, I forgot that data.ns.d can also contain offsets and I dont really know what it is. It looks like that: { plot_4: -5, plot_5: -5 }. And we can see on pine-facade that plot_4 and plot_5 are the only two plots that have the type 'shapes'.

[Mathieu2301]

Logically the x, x1, x2 values should be the index of an item in data.ns.indexes.

So I tried with Divergence for Many v4 on a daily BINANCE:BTCEUR chart, and there is exactly 96 items in data.ns.indexes, and the last label's x2 value is 95.

So it's consistent but the data.ns.indexes array looks like that :

[
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000, -2000000, -2000000, -2000000,
  -2000000, -2000000, -2000000,        8,        9,       12,
        19,       20,       22,       30,       36,       37,
        40,       41,       51,       52,       66,       67,
]

So...? I dont know, it's weird...

[MakiBM]

Thx for picking this up so quickly!

So yeah, it seems we arrived at similar place. This is were I mentioned I'd need some guidance on indexing, hoping that maybe you were dealing with that before. Seems we're both lost here - but I'm happy to brainstorm on it and do further research, we might end up with something together...

So what I'm thinking is that those lines, labels and their indexes belong to graphic commands and the huge negative number is there to keep those drawing out of computation. It looks like positive numbers show up for the lines you see on the screen.

What I need to doublecheck is that assumption - 0 index is first candle from the left that you see on the screen so reapplying timestamp gets trickier here.

Then I was thinking that it could be possible to get the rest of the drawing by scrolling the chart to the left. It seems to be a limit per call but you can do multiple calls that would load more data to the left

These messages are being sent when you scroll to the left or zoom out the chart:

At this point we get timescale_update with some info. I arrived at this - not sure how to merge it together... Will keep debugging this and hopefully come out with something creative I could share.

Side note - with all that hassle above I was wondering if this is the best way to handle that. I went another rabbit hole here - trying to find on a window object if there is any storage they use for everything that arrives from sockets. I would expect storage to be organised already and they handle heavy lifting with merging the partial data that arrives. I was no lucky finding that exposed. But I just learnt on queryObjects - that might give an opportunity to find what's there even if not exposed. Have you ever tried this path? My reasoning is that this could be easier to read normalised state

BTW - great job on v3. I've just started to dig into the guts of TV myself and found your repo very helpful. I got here when I started to have the issues above so lots of stuff I've handled on my own with mocking up stuff here and there to get what I need. Hopefully I can bring something back to your project

[MakiBM]

This is more offtopic but in case it would be helpful in any way.

So I went with the queryObject rabbit hole a bit further and it seems they actually do keep everything in one place. When I scroll the chart to the first available candle is the same I have here on the shots and it looks they appy some normalisation and we have now both index and timestamp

[Mathieu2301]

Okayy, so you found this in keyvaluepairs indexed DB ? And how do you plan to implement this ? With a headless browser like Puppeteer ?

The initial goal of this project is to imitate the behavior of TradingView, not simulate it 🤔 Adding a headless browser to the library will make it very heavy when we could rewrite the behavior of the website. It is also better for stability as the TradingView website may eventually change but normally the websocket api should stay the same.

[Mathieu2301]

In any cases, I start working on the pinescript indicator tomorrow, it will be useful to us in all cases :)

[MakiBM]

Yeah, I guess you're right - you have a library to develop there and the path is clear. I'm just researching for the easiest way to get to the TV data programatically. Been researching the browser instance to see if I can get the data that is already formatted and organised. I don't mind puppeteer for what I'm doing, but totally agree on your approach for your job there. I'm sharing findings in case we overlap somewhat and this could help...

As for what I found. I've been doing queryObjects(Array) and queryObjects(Object) to find all primitives instances in not exposed scopes and filter through them to see if there is a central store that keeps all data. I'm still digging that hole. What I'm hoping for is a central store and an object reference to it. At that point I could reuse in puppeteer a browser session and control TV (load more etc) through sockets similarly to what you do in the lib. Rather than managing sockets return messages on my own I would just use a js reference to the central store and let that data be organised by the TV js. I wouldn't care on their UI there and could adjust to their changes quite easily.

[Mathieu2301]

Hello, I just published my work in my last commit. I managed to stably work with indexes and storage (manage the create / erase instructions) ! 😁 It remains to make the parser (src\chart\graphicParser.js) and it will be done ! 😉

[MakiBM]

That's great to hear, congrats! I got stuck within the puppeteer approach, seems like a dead end. Issues totally outside of the TV itself...

I'm crossing fingers for you :) Will try to review your latest stuff and see if I can contribute somewhat

[Mathieu2301]

I just found new types of data : hhists, horizlines and polygons. There are provided by built-in volume-based indicators. I don't know what hhists could mean. I currently don't have the time to work on the project but I know that it will be easy to parse all these types because we just have to translate the keys (abbreviations like t, sz, st) to understandable words (like title, size, style) and do all the JSDoc.

[jppeltier]

Hi there, thanks Mathieu for pointing me back here.

I apparently reached the same questions as both of you. But I am unfortunately far from having your Pine Code and Web Sockets hacking skills.

I may just point out two things, after reading the thread:

• graphic objects indexes (x, x1, x2) start at 0, but 0 is not necessarily the very first candle (see discussion #18); and
• I guessed ci stands for Colour Index, and hhists for Horizontal Histograms (maybe the volume profiles?), I otherwise agree with the rest of your assumptions above.

Let me know if I can be of any further assistance. I'll keep on following here.

[Mathieu2301]

Hey @MakiBM @jppeltier ! I just pushed a stable version of the GraphicParser. This commit marks the release of v3.0 ! 🥳 Can you tell me if you still have problems with this module?

[jppeltier]

I have undefined x, x1, and x2.

[Mathieu2301]

For which indicator and timeframe ?

[Mathieu2301]

@jppeltier If you have undefined values, you should increase your range value. Please set it to something like 10000 and tell me if it works.

[jppeltier]

chart.setMarket('BINANCE:BTCUSDT', {
  timeframe: 'D',
});
[...]
TradingView.getIndicator('STD;Zig_Zag').then((indic) => {

It's exactly the same one as previously, when I was getting ordered series of indexes x, x1, and x2.

I've tried with range = 10000 and it seems to be working now, congratulations!

[Mathieu2301]

Nice ! I published a working example here.

[MakiBM]

Great job Mathieu!