search

wafels / hv-analysis

Analysis tools for understanding Helioviewer Project usage
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

How to use that #1

Open PabloA-dev-coder opened 6 years ago

PabloA-dev-coder commented 6 years ago

Hello, I am trying to use those scripts to provide info for helioviewer.ias.u-psud.fr How to start with it ?

Best regards, Pablo IAS

wafels commented 6 years ago

Oyabun,

The files that contain "_prepare" convert CSV files to pickle files that are loaded by the "analyze" routines. Let me check on the format of the CSV files.

jack

On Wed, May 2, 2018 at 11:47 AM, Oyabun notifications@github.com wrote:

Hello, I am trying to use those scripts to provide info for helioviewer.ias.u-psud.fr How to start with it ?

Best regards, Pablo IAS

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/wafels/hv-analysis/issues/1, or mute the thread https://github.com/notifications/unsubscribe-auth/AA8CF8_ePkMtNc-LrUDKIAmIeZZmGktZks5tudT3gaJpZM4TvtBG .

PabloA-dev-coder commented 6 years ago

Hello Wafels ;-), What about the CSV files that you mentioned , are they an output of the database, can you transmit the request if I am right ? So I'll try to reproduce that scenario with our instance helioviewer.ias.u-psud.fr

We would love to have a readme.txt file to start

P. (IAS)

wafels commented 6 years ago

The information should be pretty easy to get out of the database as a CSV file

The screenshot information needed is

id - timestamp - ObservationDate - DataSourceID

The HV movie generation information needed is

id - timestamp - StartDate - EndDate - DataSourceID The JHV movie generation information needed is

movieId - timestamp I don't have a record of the time-range of data that JHV users request.

Let me know how you get on. I'm preparing some plots for the poster. I'm going to print the poster on May 17.

On Mon, May 7, 2018 at 10:51 AM, Oyabun notifications@github.com wrote:

Hello Wafels ;-), What about the CSV files that you mentioned , are they an output of the database, can you transmit the request if I am right ? So I'll try to reproduce that scenario with our instance helioviewer.ias.u-psud.fr

We would love to have a readme.txt file to start

P. (IAS)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wafels/hv-analysis/issues/1#issuecomment-387090182, or mute the thread https://github.com/notifications/unsubscribe-auth/AA8CFy-UEYeMbGSbIxev_wHj17nSExODks5twF91gaJpZM4TvtBG .

PabloA-dev-coder commented 6 years ago

Hello Jack ,

Here are the description of the table screenshots , movies moviesFormats 

mysql> describe screenshots ;
+-------------------+---------------------+------+-----+-------------------+----------------+
| Field             | Type                | Null | Key | Default           | Extra          |
+-------------------+---------------------+------+-----+-------------------+----------------+
| id                | int(10) unsigned    | NO   | PRI | NULL              | auto_increment |
| timestamp         | timestamp           | NO   |     | CURRENT_TIMESTAMP |                |
| observationDate   | datetime            | NO   |     | NULL              |                |
| imageScale        | float unsigned      | YES  |     | NULL              |                |
| regionOfInterest  | polygon             | NO   |     | NULL              |                |
| watermark         | tinyint(1) unsigned | YES  |     | 1                 |                |
| dataSourceString  | varchar(255)        | NO   |     | NULL              |                |
| dataSourceBitMask | bigint(20) unsigned | YES  |     | NULL              |                |
| eventSourceString | varchar(1024)       | YES  |     | NULL              |                |
| eventsLabels      | tinyint(1) unsigned | NO   |     | NULL              |                |
| movieIcons        | tinyint(1) unsigned | NO   |     | 0                 |                |
| scale             | tinyint(1) unsigned | NO   |     | 0                 |                |
| scaleType         | varchar(12)         | YES  |     | earth             |                |
| scaleX            | float               | YES  |     | 0                 |                |
| scaleY            | float               | YES  |     | 0                 |                |
| numLayers         | tinyint(3) unsigned | NO   |     | 1                 |                |
+-------------------+---------------------+------+-----+-------------------+----------------+
16 rows in set (0.00 sec)

mysql> describe movies ;

+--------------------+----------------------+------+-----+---------------------+----------------+
| Field              | Type                 | Null | Key | Default             | Extra          |
+--------------------+----------------------+------+-----+---------------------+----------------+
| id                 | int(10) unsigned     | NO   | PRI | NULL                | auto_increment |
| timestamp          | timestamp            | NO   |     | CURRENT_TIMESTAMP   |                |
| reqStartDate       | datetime             | NO   |     | NULL                |                |
| reqEndDate         | datetime             | NO   |     | NULL                |                |
| reqObservationDate | datetime             | YES  |     | NULL                |                |
| imageScale         | float                | NO   |     | NULL                |                |
| regionOfInterest   | polygon              | NO   |     | NULL                |                |
| maxFrames          | smallint(6)          | NO   |     | NULL                |                |
| watermark          | tinyint(1) unsigned  | NO   |     | NULL                |                |
| dataSourceString   | varchar(255)         | NO   |     | NULL                |                |
| dataSourceBitMask  | bigint(20) unsigned  | YES  |     | NULL                |                |
| eventSourceString  | varchar(1024)        | YES  |     | NULL                |                |
| eventsLabels       | tinyint(1) unsigned  | NO   |     | NULL                |                |
| movieIcons         | tinyint(1) unsigned  | NO   |     | 0                   |                |
| followViewport     | tinyint(1)           | YES  |     | 0                   |                |
| scale              | tinyint(1) unsigned  | NO   |     | 0                   |                |
| scaleType          | varchar(12)          | YES  |     | earth               |                |
| scaleX             | float                | YES  |     | 0                   |                |
| scaleY             | float                | YES  |     | 0                   |                |
| numLayers          | tinyint(3) unsigned  | YES  |     | NULL                |                |
| queueNum           | smallint(5) unsigned | YES  |     | NULL                |                |
| frameRate          | float unsigned       | YES  |     | NULL                |                |
| movieLength        | float unsigned       | YES  |     | NULL                |                |
| startDate          | datetime             | YES  | MUL | NULL                |                |
| endDate            | datetime             | YES  | MUL | NULL                |                |
| numFrames          | smallint(5) unsigned | YES  |     | NULL                |                |
| width              | smallint(5) unsigned | YES  |     | NULL                |                |
| height             | smallint(5) unsigned | YES  |     | NULL                |                |
| buildTimeStart     | timestamp            | NO   |     | 0000-00-00 00:00:00 |                |
| buildTimeEnd       | timestamp            | NO   |     | 0000-00-00 00:00:00 |                |
| size               | tinyint(2)           | NO   |     | 0                   |                |
+--------------------+----------------------+------+-----+---------------------+----------------+
31 rows in set (0.01 sec)

for the table moviesFormats mysql> describe movieFormats ; +----------+----------------------+------+-----+-------------------+----------------+ | Field | Type | Null | Key | Default | Extra | +----------+----------------------+------+-----+-------------------+----------------+ | id | int(10) unsigned | NO | PRI | NULL | auto_increment | | movieId | int(10) unsigned | NO | MUL | NULL | | | format | varchar(255) | NO | MUL | NULL | | | status | varchar(255) | NO | | NULL | | | procTime | smallint(5) unsigned | YES | | NULL | | | modified | timestamp | YES | | CURRENT_TIMESTAMP | | +----------+----------------------+------+-----+-------------------+----------------+ 6 rows in set (0.00 sec)

I don't have DataSourceID sorry , has it been added recently? Can I use dataSourceString instead ?

You will find attached the csv files Maybe you will be able to use them I have problems to run your code (I miss /Data/hvanalysis/source/getDataSources.json for ex)

ias_jhv_movies.txt screenshots.txt movies.txt

wafels commented 6 years ago

I took a quick look at the files. It seems like most of the results are from 2018. Do you have any stats from earlier?

thanks!

On Tue, May 15, 2018 at 11:55 AM, Oyabun notifications@github.com wrote:

Hello Jack ,

Here are the description of the table screenshots , movies moviesFormats

mysql> describe screenshots ; +-------------------+---------------------+------+-----+-------------------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------------+---------------------+------+-----+-------------------+----------------+ | id | int(10) unsigned | NO | PRI | NULL | auto_increment | | timestamp | timestamp | NO | | CURRENT_TIMESTAMP | | | observationDate | datetime | NO | | NULL | | | imageScale | float unsigned | YES | | NULL | | | regionOfInterest | polygon | NO | | NULL | | | watermark | tinyint(1) unsigned | YES | | 1 | | | dataSourceString | varchar(255) | NO | | NULL | | | dataSourceBitMask | bigint(20) unsigned | YES | | NULL | | | eventSourceString | varchar(1024) | YES | | NULL | | | eventsLabels | tinyint(1) unsigned | NO | | NULL | | | movieIcons | tinyint(1) unsigned | NO | | 0 | | | scale | tinyint(1) unsigned | NO | | 0 | | | scaleType | varchar(12) | YES | | earth | | | scaleX | float | YES | | 0 | | | scaleY | float | YES | | 0 | | | numLayers | tinyint(3) unsigned | NO | | 1 | | +-------------------+---------------------+------+-----+-------------------+----------------+ 16 rows in set (0.00 sec)

mysql> describe movies ;

+--------------------+----------------------+------+-----+---------------------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------------+----------------------+------+-----+---------------------+----------------+ | id | int(10) unsigned | NO | PRI | NULL | auto_increment | | timestamp | timestamp | NO | | CURRENT_TIMESTAMP | | | reqStartDate | datetime | NO | | NULL | | | reqEndDate | datetime | NO | | NULL | | | reqObservationDate | datetime | YES | | NULL | | | imageScale | float | NO | | NULL | | | regionOfInterest | polygon | NO | | NULL | | | maxFrames | smallint(6) | NO | | NULL | | | watermark | tinyint(1) unsigned | NO | | NULL | | | dataSourceString | varchar(255) | NO | | NULL | | | dataSourceBitMask | bigint(20) unsigned | YES | | NULL | | | eventSourceString | varchar(1024) | YES | | NULL | | | eventsLabels | tinyint(1) unsigned | NO | | NULL | | | movieIcons | tinyint(1) unsigned | NO | | 0 | | | followViewport | tinyint(1) | YES | | 0 | | | scale | tinyint(1) unsigned | NO | | 0 | | | scaleType | varchar(12) | YES | | earth | | | scaleX | float | YES | | 0 | | | scaleY | float | YES | | 0 | | | numLayers | tinyint(3) unsigned | YES | | NULL | | | queueNum | smallint(5) unsigned | YES | | NULL | | | frameRate | float unsigned | YES | | NULL | | | movieLength | float unsigned | YES | | NULL | | | startDate | datetime | YES | MUL | NULL | | | endDate | datetime | YES | MUL | NULL | | | numFrames | smallint(5) unsigned | YES | | NULL | | | width | smallint(5) unsigned | YES | | NULL | | | height | smallint(5) unsigned | YES | | NULL | | | buildTimeStart | timestamp | NO | | 0000-00-00 00:00:00 | | | buildTimeEnd | timestamp | NO | | 0000-00-00 00:00:00 | | | size | tinyint(2) | NO | | 0 | | +--------------------+----------------------+------+-----+---------------------+----------------+ 31 rows in set (0.01 sec)

for the table moviesFormats mysql> describe movieFormats ; +----------+----------------------+------+-----+------------ -------+----------------+ | Field | Type | Null | Key | Default | Extra | +----------+----------------------+------+-----+------------ -------+----------------+ | id | int(10) unsigned | NO | PRI | NULL | auto_increment | | movieId | int(10) unsigned | NO | MUL | NULL | | | format | varchar(255) | NO | MUL | NULL | | | status | varchar(255) | NO | | NULL | | | procTime | smallint(5) unsigned | YES | | NULL | | | modified | timestamp | YES | | CURRENT_TIMESTAMP | | +----------+----------------------+------+-----+------------ -------+----------------+ 6 rows in set (0.00 sec)

I don't have DataSourceID sorry , has it been added recently? Can I use dataSourceString instead ?

You will find attached the csv files Maybe you will be able to use them I have problems to run your code (I miss /Data/hvanalysis/source/getDataSources.json for ex)

ias_jhv_movies.txt https://github.com/wafels/hv-analysis/files/2005690/ias_jhv_movies.txt screenshots.txt https://github.com/wafels/hv-analysis/files/2005696/screenshots.txt movies.txt https://github.com/wafels/hv-analysis/files/2005697/movies.txt

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wafels/hv-analysis/issues/1#issuecomment-389218848, or mute the thread https://github.com/notifications/unsubscribe-auth/AA8CF1iArKVTZf4pqOQI2e1x-GkER3hqks5tyvpXgaJpZM4TvtBG .

PabloA-dev-coder commented 6 years ago

Hello Jack , We don't have stats before 2018 sorry.

Pablo

ebuchlin commented 6 years ago

We have looked in a database dump from 2017-10-13 (with older stats), there are no columns DataSourceID or DataSourceNames (as required by hvorg_movies_prepare.py). We are also missing ~/Data/hvanalysis/source/getDataSources.json .