Looking forward to 2021

wetterfrosch commented 3 years ago

a little collection/notes on stuff trying to get things towards grafana 7:

six v1.13.0

Successfully built grafanimate
ERROR: mozrunner 8.2.1 has requirement six<2,>=1.13.0, but you'll have six 1.11.0 which is incompatible.
ERROR: mozlog 7.1.0 has requirement six>=1.13.0, but you'll have six 1.11.0 which is incompatible.

seems like within ubuntu/debilian's pip-world, mozrunner and mozlog require six >=1.13.0. changing to that version in setup.py helped.

`<grafana-app>`

/seems/ to me that waiting for fails. (it is still there in grafana7).

list of panels as promises not aquired

when replacing the awaiting of with a little sleep, we get here: 2021-11-13 22:45:35,661 [grafanimate.grafana ] INFO : Waiting for "all-data-received" event and go timeout.

all-data-received seems to wait for some promises in grafana-studio.js -- after uncommenting log('panel:', panel); it seems that the dashboard-list is already empty. turtles! turtles everywhere! :)

grafanimate --grafana-url=https://play.grafana.org/ --dashboard-uid=000000012 --scenario=wtf_demo
2021-11-13 23:20:13,854 [grafanimate.grafana           ] INFO   : Starting GrafanaWrapper on https://play.grafana.org/
2021-11-13 23:20:13,854 [grafanimate.marionette        ] INFO   : Starting Marionette Gecko wrapper
2021-11-13 23:20:13,855 [grafanimate.marionette        ] INFO   : Found "firefox" program at /usr/bin/firefox
2021-11-13 23:20:13,855 [grafanimate.marionette        ] INFO   : Check for running instance of Marionette/Firefox at localhost:2828
2021-11-13 23:20:13,859 [grafanimate.marionette        ] INFO   : Will reuse running Marionette/Firefox
2021-11-13 23:20:13,878 [grafanimate.grafana           ] INFO   : Starting Grafana at https://play.grafana.org/
2021-11-13 23:20:16,095 [grafanimate.grafana           ] INFO   : Waiting for Grafana to load
2021-11-13 23:20:19,644 [grafanimate.grafana           ] INFO   : Grafana loaded
2021-11-13 23:20:19,676 [grafanimate.animations        ] INFO   : Opening dashboard
2021-11-13 23:20:19,677 [grafanimate.grafana           ] INFO   : Running Javascript: return grafanaStudio.openDashboard("000000012", {"dashboard-view": null, "panel-id": null, "datetime-format": null, "header-layout": ["large-font"]});
2021-11-13 23:20:19,768 [grafanimate.grafana           ] INFO   : Waiting for "all-data-received" event
2021-11-13 23:20:39,794 [grafanimate.grafana           ] WARNING: Timed out waiting for data: Timed out after 20.0 seconds. Continuing anyway.
2021-11-13 23:20:39,796 [grafanimate.animations        ] INFO   : Dashboard ready
2021-11-13 23:20:39,796 [grafanimate.scenarios         ] INFO   : Running wtf demo
2021-11-13 23:20:39,799 [grafanimate.animations        ] INFO   : Creating rrule: dtstart=2021-11-12 09:00:00, until=2021-11-12 10:00:00, freq=5, interval=10
2021-11-13 23:20:39,802 [grafanimate.animations        ] INFO   : Datetime step: 2021-11-12 09:00:00
2021-11-13 23:20:39,803 [grafanimate.grafana           ] INFO   : Timewarp to 2021-11-12 09:00:00 -> 2021-11-12 09:09:59
2021-11-13 23:20:40,058 [grafanimate.grafana           ] INFO   : Waiting for "all-data-received" event

wetterfrosch commented 3 years ago

Wait, wait! The grafana-app-element problem doesn't seem to exist with another grafana, v6, instance. example above is with play.grafana.com, v7 -- and probably slightly different style than grafana vanilla...) ... confusing... I will try some more things ...

amotl commented 3 years ago

Hi,

thank you for reporting this. We started to give this program some love on behalf of #4. Specifically, fec675da improves the event handling in grafana-studio.js by using an adjusted waitForDashboard method.

You can check all the improvements by invoking git fetch and git checkout modernize in your working tree. Please note that it is a thorough modernization, also switching to Python 3, so you probably need to bounce your virtualenv.

With kind regards, Andreas.

wetterfrosch commented 3 years ago

Thanks!

While trying under an recent Ubuntu I get some NameError: name 'basestring' is not defined within marionette/gecko:

$ grafanimate --grafana-url=https://weather.hiveeyes.org/ --dashboard-uid=start --scenario=ctsb_tomorrow
2021-11-14 11:35:49,605 [grafanimate.grafana           ] INFO   : Starting GrafanaWrapper on https://weather.hiveeyes.org/
2021-11-14 11:35:49,605 [grafanimate.marionette        ] INFO   : Starting Marionette Gecko wrapper
2021-11-14 11:35:49,605 [grafanimate.marionette        ] INFO   : Found "firefox" program at /usr/bin/firefox
2021-11-14 11:35:49,605 [grafanimate.marionette        ] INFO   : Check for running instance of Marionette/Firefox at localhost:2828
2021-11-14 11:35:49,606 [grafanimate.marionette        ] INFO   : Will launch new Marionette/Firefox instance
mozversion INFO | application_buildid: 20211028161635
mozversion INFO | application_changeset: feceab03c6ff8b7ce7e52e759ddb6a23b310a9e5
mozversion INFO | application_display_name: Firefox
mozversion INFO | application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
mozversion INFO | application_name: Firefox
mozversion INFO | application_remotingname: firefox
mozversion INFO | application_repository: https://hg.mozilla.org/releases/mozilla-release
mozversion INFO | application_vendor: Mozilla
mozversion INFO | application_version: 94.0
mozversion INFO | platform_buildid: 20211028161635
mozversion INFO | platform_changeset: feceab03c6ff8b7ce7e52e759ddb6a23b310a9e5
mozversion INFO | platform_repository: https://hg.mozilla.org/releases/mozilla-release
mozversion INFO | platform_version: 94.0
Traceback (most recent call last):
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/bin/grafanimate", line 8, in <module>
    sys.exit(run())
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/grafanimate/commands.py", line 110, in run
    grafana = make_grafana(options['grafana-url'])
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/grafanimate/core.py", line 16, in make_grafana
    grafana.boot_firefox(headless=False)
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/grafanimate/marionette.py", line 78, in boot_firefox
    self.marionette = Marionette(
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/marionette_driver/marionette.py", line 616, in __init__
    self.start_binary(self.startup_timeout)
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/marionette_driver/marionette.py", line 630, in start_binary
    self.instance.start()
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/marionette_driver/geckoinstance.py", line 300, in start
    self._update_profile(self.profile)
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/marionette_driver/geckoinstance.py", line 215, in _update_profile
    if isinstance(profile_path, basestring):
NameError: name 'basestring' is not defined
Exception ignored in: <function Marionette.__del__ at 0x7f8fd09f2940>
Traceback (most recent call last):
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/marionette_driver/marionette.py", line 659, in __del__
    self.cleanup()
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/marionette_driver/marionette.py", line 653, in cleanup
    self.instance.close(clean=True)
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/marionette_driver/geckoinstance.py", line 353, in close
    self.profile = None
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/marionette_driver/geckoinstance.py", line 193, in profile
    self._update_profile(value)
  File "/home/wtf/noc/ctsb/grafanimate/.venv3/lib/python3.8/site-packages/marionette_driver/geckoinstance.py", line 215, in _update_profile
    if isinstance(profile_path, basestring):
NameError: name 'basestring' is not defined

wetterfrosch commented 3 years ago

venv was created this way:

virtualenv --python=python3 .venv3
source .venv3/bin/activate
pip install grafanimate-main/

No errors within installation (after switching six to v1.13.0). [edit: python3 introduces itself as Python 3.8.10]

amotl commented 3 years ago

Hm, it looks like it might be using an older version of marionette_driver. I bumped it to marionette_driver>=3,<4 within the setup.py file.

With an activated virtualenv, what does this tell you on your machine?

$ python
Python 3.9.7 (default, Oct 13 2021, 06:44:56)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import marionette_driver
>>> marionette_driver.__version__
'3.1.0'

P.S.: six is not explicitly part of the list of dependencies anymore, however, both marionette_driver and python-dateutil will pull it in again.

My marionette_driver/geckodriver.py file shows at lines 223 ff., where your stacktrace was showing if isinstance(profile_path, basestring) on a different line:

# If a path to a profile is given then clone it
if isinstance(profile_path, six.string_types):
    profile_args["path_from"] = profile_path
    profile_args["path_to"] = tempfile.mkdtemp(
        suffix=u".{}".format(profile_name or os.path.basename(profile_path)),
        dir=self.workspace)
    # The target must not exist yet
    os.rmdir(profile_args["path_to"])

    profile = Profile.clone(**profile_args)

basestring is nothing you would expect to find within sane Python 3 programs.

The built-in basestring abstract type was removed. Use str instead. The str and bytes types don’t have functionality enough in common to warrant a shared base class. The 2to3 tool (see below) replaces every occurrence of basestring with str.

-- https://docs.python.org/3/whatsnew/3.0.html

amotl commented 3 years ago

Maybe invoke, within the activated virtualenv again, pip install --editable=. --upgrade.

amotl commented 3 years ago

I just verified it on Python 3.8.12 as well. Everything works on my machine. Maybe start from scratch by freshly cloning the repository?

git clone https://github.com/panodata/grafanimate --branch=modernize
cd grafanimate
make test

source .venv/bin/activate
time grafanimate --grafana-url=https://swarm.hiveeyes.org/grafana/ --scenario=ir_sensor_svg_pixmap --dashboard-uid=acUXbj_mz --header-layout=studio --datetime-format=human-time --debug

P.S.: For whatever reason, on Python 3.8, the process stops at the end, like

2021-11-14 12:20:57,502 [grafanimate.commands          ] INFO   : Produced 2 results
[
  "./var/results/labor-ir-sensor-svg-pixmap.mp4",
  "./var/results/labor-ir-sensor-svg-pixmap.gif"
]

You will have to hit CTRL+C once, without any harm. It does not happen on Python 3.9 to me.

grafana-toolbox / grafanimate