search

nostr-protocol / nips

Nostr Implementation Possibilities
2.38k stars 577 forks source link

NIP-XX Use Nostr as storage for chart data #743

Open ShinoharaTa opened 1 year ago

ShinoharaTa commented 1 year ago

Use Nostr as storage for chart data

The method I propose is to extend NIP-78 (KIND:30078) and define it as a chart. That mechanism in Nostr is very resistant to communication failures and also overcomes cloud vendor failures.
We believe that this will be of great use in Japan, where there are many disasters, and we believe that by placing preliminary data, etc. on a high-speed distributed network, open data can be delivered reliably.

For your reference, I propose a time series of data that we Japanese see every time a disaster strikes.

Of course, you can use a variety of observation data, not just weather data.
Here are some familiar examples.

Although different from a chart, we think it would be useful to store disaster information in a relay.
This should be suggested in another NIP.

As an example of using NIP-78, I run the following website and contribute observation data to the relay.

野州田川水系 定点観測所 - * japanese only.

It measures the number of KIND: 1's posted to relays operated by Japanese and stores them in KIND: 30078.
The measured data is used as a chart like the water level of a river.

Open data can be viewed by anyone, and if we can define it as a Chart event, we can provide a system that makes open data acquisition much easier.

I propose a time series data structure, extending the tags of NIP-78 (Kind: 30078) as follows.

Chart structure as I define it

This is a sample and objects CPU usage, RAM usage, and CPU temperature for 12 hours. KIND: 30078 as well as the d tag and AUTHOR to determine.
You may want to overlay multiple data on a single chart. The number of values can be added as desired so that multiple data can be registered.

{
    "kind": 30076,
    "author": "npub",
    "tags": [
        ["d", "chart_identification_code"],
        ["title", "chart Title"],
        ["category", "Server", "Temperature"],
        ["description", ""],
        ["license", ""],
        ["published_at", "ISO Unix Time String"],
        [
            // dataset, "data set name", "x label", "y label", "data as a JSON string of x,y pairs"
            "dataset", "CPU Usage", "Date", "Percentage", "[{x: 1690862400, y: 12.5},{x:.., y:..}]"
        ],
        [
            "dataset", "RAM Usage", "Date", "Percentage", "[{x: 1690862400, y: 45},{x:.., y:..}]"
        ],
        [
            "dataset", "CPU Temp", "Date", "Temperature (C)", "[{x: 1690862400, y: 65},{x:.., y:..}]"
        ],
    ]
}

We believe that labels and values properly extracted from this data structure can be passed directly to the Chart.js object as an example.
If anyone can provide open data to the site in an easy-to-read format, it will be ready for disasters and cloud failures.

The data defined for this purpose is defined as in NIP-19, nchart1xxxxxx... We believe that it would be more convenient to use if the data defined in this way could be defined as NIP-19 and provided as

Constraint

This proposal places constraints on existing events. Relay load should also be considered, especially when using large data. Define constraints.

Some data may be retained for a long period of time. In that case, add a date to the d tag as shown below. The date of interest is controlled by the date tag, which should be written in unixtime. If this is data that is being collected continuously, the date tag is not necessary. Define that the data is always backward from the current time.

{
    "kind": 30076,
    "author": "npub",
    "tags": [
        ["d", "chart_identification_code_yyyymmdd"],
        ["title", "chart Title"],
        ["category", "Server", "Temperature"],
        ["description", ""],
        ["date", "ISO Unix Time String"],
        ["license", ""],
        ["published_at", "ISO Unix Time String"],
        [
            "dataset", "CPU Usage", "Date", "Percentage", "[{x: 1690862400, y: 12.5},{x:.., y:..}]"
        ],
        [
            "dataset", "RAM Usage", "Date", "Percentage", "[{x: 1690862400, y: 45},{x:.., y:..}]"
        ],
        [
            "dataset", "CPU Temp", "Date", "Temperature (C)", "[{x: 1690862400, y: 65},{x:.., y:..}]"
        ],
    ]
}

Data concatenation

When the number of data becomes large, you can manage multiple data by concatenating each POSTS. This will use the NIP-51 list.

{
    "kind": 3007x,
    "author": "npub",
    "tags": [
        ["d", "chart_identification_code"],
        ["title", "chart Title"],
        ["category", "Server", "Temperature"],
        ["description", ""],
        ["license", ""],
        ["published_at", "ISO Unix Time String"],
        ["a", "30076:fcaec..d590f:abcd"],
        ["a", "30076:fcaec..d590f:efgh"],
        ["a", "30076:fcaec..d590f:ijkl"],
        ...
    ]
}
vitorpamplona commented 1 year ago

No need to change 30078, you can just create a new event type for this application.

decentropy commented 1 year ago

@ShinoharaTa

I dont think you are describing a graph... more like a table, right? graphs have nodes and edges.

im actually working on web apps for both cases. Im using vis.js to display graph, and handsontable for table.

DM me on nostr if you want to collaborate

ShinoharaTa commented 1 year ago

@decentropy Indeed, this is not based on graph theory. The proposal is ngraph... I will change the proposal from "ngraph..." to "chart" and propose it again.

mattn commented 1 year ago

how about IDs for next/previous link?

mattn commented 1 year ago

I suggest that you change this suggestion to make a standard time series data format. Chart is one of the way to display the data. Someone might way to use analyzing.

The elements required for a time series data format are (I think):

vitorpamplona commented 1 year ago

Suggestion for tag structure: 

{
    "kind": 30075,
    "author": "npub",
    "tags": [
        ["d", "chart_identification_code"],
        ["title", "chart Title"],
        ["category", "Server", "Temperature"],
        ["description", ""]
        ["published_at", "ISO Unix Time String"], // original date, the updated at is already the `.created_at`
        ["labels",
            "2023-08-01 00:00", "2023-08-01 01:00", "2023-08-01 02:00", "2023-08-01 03:00", 
            "2023-08-01 04:00", "2023-08-01 05:00", "2023-08-01 06:00", "2023-08-01 07:00", 
            "2023-08-01 08:00", "2023-08-01 09:00", "2023-08-01 10:00", "2023-08-01 11:00"
        ],
        [
            "dataset", "CPU Usage", "Percentage" // dataset, name, y-axis label, n-values
            "12.5%", "12.7%", "12.3%", "13.0%",
            "13.5%", "12.9%", "12.8%", "12.6%",
            "13.2%", "13.1%", "12.4%", "12.9%"
        ],
        [
            "dataset", "RAM Usage", "Percentage"
            "45%", "46%", "44%", "45%",
            "47%", "46%", "45%", "44%",
            "46%", "47%", "44%", "45%"
        ],
        [
            "dataset",  "CPU Temp", "Temperature (C)"
            "65°C", "66°C", "64°C", "65°C",
            "66°C", "65°C", "64°C", "64°C",
            "65°C", "66°C", "64°C", "65°C"
        ]
    ]
}

This event kind is ideal for certain charts, where xlabel can be the same (pre-rounded) for all datasets.

Another event kind could be ideal for more granular data that can be rounded at the client side. 

{
    "kind": 30076,
    "author": "npub",
    "tags": [
        ["d", "chart_identification_code"],
        ["title", "chart Title"],
        ["category", "Server", "Temperature"],
        ["description", ""]
        ["published_at", "ISO Unix Time String"],   
        [
            // dataset, "data set name", "x label", "y label", "data as a JSON string of x,y pairs"
            "dataset", "CPU Usage", "Date", "Percentage", "[{x: 1690862400, y: 12.5},{x:.., y:..}]"
        ],
        [
            "dataset", "RAM Usage", "Date", "Percentage", "[{x: 1690862400, y: 45},{x:.., y:..}]"
        ],
        [
            "dataset", "CPU Temp", "Date", "Temperature (C)", "[{x: 1690862400, y: 65},{x:.., y:..}]"
        ],
    ]
}
ShinoharaTa commented 1 year ago

@vitorpamplona Thank you very much. That suggestion is very simple and straightforward. I will revise my proposal.

ShinoharaTa commented 1 year ago

@mattn We have addressed two of your suggestions.

For the life of me, I could not decide on the following. We think it would be better to concatenate the datasets in KIND, which concatenates multiple Charts.

Can continuous data be defined as follows...

{
    "kind": 3007x,
    "author": "npub",
    "tags": [
        ["d", "chart_identification_code"],
        ["title", "chart Title"],
        ["category", "Server", "Temperature"],
        ["description", ""],
        ["license", ""],
        ["published_at", "ISO Unix Time String"],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."] 
    ]
}
ShinoharaTa commented 1 year ago

MEMO.

chart data kind: 30078 -> 30076, concat data kind: 30078 -> 3007x,

AsaiToshiya commented 1 year ago

Could you use l, L and a of standardized tags instead of category and chart tag?

The data defined for this purpose is defined as in NIP-19, nchart1xxxxxx... We believe that it would be more convenient to use if the data defined in this way could be defined as NIP-19 and provided as

What is included in TLV of nchart1...? If it is just a marker, I think nevent1... is enough.

{
    "kind": 3007x,
    "author": "npub",
    "tags": [
        ["d", "chart_identification_code"],
        ["title", "chart Title"],
        ["category", "Server", "Temperature"],
        ["description", ""],
        ["license", ""],
        ["published_at", "ISO Unix Time String"],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."],
        ["chart", "nchart1..."] 
    ]
}

It looks like a NIP-51 parameterized replaceable list event.

ShinoharaTa commented 1 year ago

@AsaiToshiya

I would like an example of what the events should look like if we change the tags and use the standardized l, L, a.

The reason for defining this as nchart1 instead of nevent1 is to make it clear that this event is chart data according to the format. The purpose is to fix the shape of the data for use as a chart. We believe that providing a standardized chart reference method is highly reusable for websites, applications, and other batches, and is a way to effectively utilize the fault-tolerant performance of Nostr's distributed network. If we define it as nevent1 and treat it as a NIP-51 list, how can we determine that this is chart data?

AsaiToshiya commented 1 year ago

I would like an example of what the events should look like if we change the tags and use the standardized l, L, a.

Here is an example:

{
    "kind": 3007x,
    "author": "npub",
    "tags": [
        ["d", "chart_identification_code"],
        ["title", "chart Title"],
        ["L", "#t"],
        ["l", "Server", "#t"],
        ["l", "Temperature", "#t"],
        ["description", ""],
        ["license", ""],
        ["published_at", "ISO Unix Time String"],
        ["a", "30076:fcaec..d590f:abcd"],
        ["a", "30076:fcaec..d590f:efgh"],
        ["a", "30076:fcaec..d590f:ijkl"],
        ...
    ]
}

Please see NIP-32 for more information on l and L tag.

The reason for defining this as nchart1 instead of nevent1 is to make it clear that this event is chart data according to the format. The purpose is to fix the shape of the data for use as a chart.

I don't know the benefit of clarifying as nchart1.... Probably user and client know it is chart data.

We believe that providing a standardized chart reference method is highly reusable for websites, applications, and other batches, and is a way to effectively utilize the fault-tolerant performance of Nostr's distributed network.

nevent1... is the same with it.

If we define it as nevent1 and treat it as a NIP-51 list, how can we determine that this is chart data?

You are right, a new kind must be defined even if the event is a NIP-51 list. So this is a matter of where to write.

vitorpamplona commented 1 year ago

Can you also add an example to what's on "a", "30076:fcaec..d590f:abcd" ?

AsaiToshiya commented 1 year ago

The data defined for this purpose is defined as in NIP-19, nchart1xxxxxx... We believe that it would be more convenient to use if the data defined in this way could be defined as NIP-19 and provided as

What is included in TLV of nchart1...? If it is just a marker, I think nevent1... is enough.

Sorry, nevent1... is mistake, it is naddr1....

Can you also add an example to what's on "a", "30076:fcaec..d590f:abcd" ?

An example is about the same as https://github.com/nostr-protocol/nips/issues/743#issue-1868238087.

{
    "kind": 30076,
    "author": "fcaec..d590f",
    "tags": [
        ["d", "abcd"],
        ["title", "chart Title"],
        ["L", "#t"],
        ["l", "Server", "#t"],
        ["l", "Temperature", "#t"],
        ["description", ""],
        ["license", ""],
        ["published_at", "ISO Unix Time String"],
        [
            // dataset, "data set name", "x label", "y label", "data as a JSON string of x,y pairs"
            "dataset", "CPU Usage", "Date", "Percentage", "[{x: 1690862400, y: 12.5},{x:.., y:..}]"
        ],
        [
            "dataset", "RAM Usage", "Date", "Percentage", "[{x: 1690862400, y: 45},{x:.., y:..}]"
        ],
        [
            "dataset", "CPU Temp", "Date", "Temperature (C)", "[{x: 1690862400, y: 65},{x:.., y:..}]"
        ],
    ]
}
vitorpamplona commented 1 year ago

Hum.. I don't actually understand why the 3007x event exists. Is it just a bundle of many charts? Couldn't that be just a NIP51 list?

ShinoharaTa commented 1 year ago

It seems that consolidated data should be managed in a nip-51 list. I will fix it.

Semisol commented 1 year ago

Why can't we use something like ["dataset", <type>, <name>, <unit>, <data...>]_ Type would be either xy or y (assuming equal spacing between datapoints). For data you would have either <x>, <y>, <x>, <y>, ... or <y>, <y>, ... depending on the type.