Tobii glasses data transfer in ROS !!

[seungyun1]

Hello, I am a university student living in Korea. I'm interested in using Tobii glasses' data, so I have a question while I was studying the ROS, so I'm asking this

What I want is to send real-time data from Tobii glasses to subscribers via ROS.

So, I would like to apply rospy to live_scene_and_gaze.py file.

This is the code that I use as a publisher in ROS.

I don't know how to combine this code with the live_scene_and_gaze.py code.

If you have any ideas, I'd appreciate it if you could let me know.

[ddetommaso]

Hi @SeungYoonLee1

good to hear new developers interested in integrating Tobii Glasses in ROS. I think it's totally doable to stream data using publishers in ROS.

For example, if you want to stream gaze position in a ROS publisher that sends string you'll simply need to encode this information as string and then pass it to the publisher.

Below you can find an example.

import time
from tobiiglassesctrl import TobiiGlassesController

import rospy
from std_msgs.msg import String

def tobii_talker():
      pub = rospy.Publisher('tobiichatter', String, queue_size=10)
      rospy.init_node('talker', anonymous=True)
      rate = rospy.Rate(100)
      tobiiglasses = TobiiGlassesController("192.168.71.50")
      tobiiglasses.start_streaming()
      time.sleep(3.0)
      while not rospy.is_shutdown():
               tobii_gp = "Gaze Position: %s " % tobiiglasses.get_data()['gp']
               rospy.loginfo(tobii_gp)
               pub.publish(tobii_gp)
               rate.sleep()
      tobiiglasses.stop_streaming()
      tobiiglasses.close()

[ddetommaso]

Hi @SeungYoonLee1 ,

any chance to make the controller works in ROS?

[seungyun1]

Thank you for your answer. Can you tell me what "any chance to make the controller works in ROS?" means?

It is the exam period of the university, so I think I can use Tobii to use the data in ROS around the end of April.

I'm sorry I couldn't try it quickly. Can I ask you again if I have something I don't know after the exam period is over?

[ddetommaso]

Hi @SeungYoonLee1 ,

I was just wondering if you had any chance to try the data streaming in ROS, cause it might be a potential interest for other Tobii Glasses users. Take your time with exams and good luck with your studies.

Keep in touch!

[seungyun1]

Yes, thank you! I haven't tried streaming data in ROS yet.

I'll do it right away when the test is over.

Keep in touch!

[seungyun1]

Hello, I came back after finishing the exam. I'm studying how to utilize data using Tobii in ROS. I created talker.py for publisher by combining the code you told me last time with the code live_scene_and_gaze.py, and added listener.py to make sure that data transfer is smooth.

After running it, I was able to see that data such as 'gp' was sent well.

talker.py and listner.py are attached below. I would appreciate it if you could let me know if you have any additional feedback.

I have additional questions about utilizing tobii data, so I'm asking you this.

I found out that it's 'cvbridge', and I want to use it to publish the frame that I'm filming in real time on the tobii camera.

I've tried many things, but I don't know how to hand over the frame to the subscriber.

I would appreciate it if you could let me know if you have any advice!

Here's the link to 'cvbridge' in ROS. http://wiki.ros.org/cv_bridge/Tutorials/ConvertingBetweenROSImagesAndOpenCVImagesPython

2021년 4월 7일 (수) 오후 6:06, Davide De Tommaso @.***>님이 작성:

Hi @SeungYoonLee1 https://github.com/SeungYoonLee1 ,

I was just wondering if you had any chance to try the data streaming in ROS, cause it might be a potential interest for other Tobii Glasses users. Take your time with exams and good luck with your studies.

Keep in touch!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ddetommaso/TobiiGlassesPyController/issues/22#issuecomment-814743968, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQU32RN6HLOEKSU4RLCJVJLTHQOBFANCNFSM4Z6GLOVA .

!/usr/bin/env python

live_scene_and_gaze.py : A demo for video streaming and synchronized gaze

#

Copyright (C) 2018 Davide De Tommaso

#

This program is free software; you can redistribute it and/or modify

it under the terms of the GNU General Public License as published by

the Free Software Foundation; either version 2 of the License, or

(at your option) any later version.

#

This program is distributed in the hope that it will be useful,

but WITHOUT ANY WARRANTY; without even the implied warranty of

MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

GNU General Public License for more details.

#

You should have received a copy of the GNU General Public License

along with this program. If not, see http://www.gnu.org/licenses/

import cv2 import numpy as np import time import rospy from std_msgs.msg import String

if hasattr(builtins, 'raw_input'): input=raw_input

from tobiiglassesctrl import TobiiGlassesController

-------------------------------------------------------------------

""" class image_converter:

def init(self): self.image_pub = rospy.Publisher("image_topic_2",Image)

self.bridge = CvBridge()
self.image_sub = rospy.Subscriber("image_topic",Image,self.callback)

def callback(self,data): try: cv_image = self.bridge.imgmsg_to_cv2(data, "bgr8") except CvBridgeError as e: print(e)

(rows,cols,channels) = cv_image.shape
if cols > 60 and rows > 60 :
  cv2.circle(cv_image, (50,50), 10, 255)

cv2.imshow("Image window", cv_image)
cv2.waitKey(3)

try:
  self.image_pub.publish(self.bridge.cv2_to_imgmsg(cv_image, "bgr8"))
except CvBridgeError as e:
  print(e)

"""

-------------------------------------------------------------------------------

def tobii_talker(): pub = rospy.Publisher('tobiichatter', String, queue_size=10) rospy.init_node('talker', anonymous=True) rate = rospy.Rate(100) tobiiglasses = TobiiGlassesController("10.2.70.109") tobiiglasses.start_streaming() time.sleep(3.0) while not rospy.is_shutdown(): tobii_gp = "Gaze Position: %s " % tobiiglasses.get_data()['gp'] rospy.loginfo(tobii_gp) pub.publish(tobii_gp) rate.sleep() tobiiglasses.stop_streaming() tobiiglasses.close()

ipv4_address = "10.2.70.109"

tobiiglasses = TobiiGlassesController(ipv4_address, video_scene=True)

project_id = tobiiglasses.create_project("Test live_scene_and_gaze.py")

participant_id = tobiiglasses.create_participant(project_id, "participant_test")

calibration_id = tobiiglasses.create_calibration(project_id, participant_id)

input("Put the calibration marker in front of the user, then press enter to calibrate") tobiiglasses.start_calibration(calibration_id)

res = tobiiglasses.wait_until_calibration_is_done(calibration_id)

if res is False: print("Calibration failed!") exit(1)

cap = cv2.VideoCapture("rtsp://%s:8554/live/scene" % ipv4_address)

Check if camera opened successfully

if (cap.isOpened()== False): print("Error opening video stream or file")

Read until video is completed

tobiiglasses.start_streaming() while(cap.isOpened()):

Capture frame-by-frame

ret, frame = cap.read()

--------------------------------------

ic = image_converter()

rospy.init_node('image_converter', anonymous=True)

rospy.spin()

-------------------------------

hms = time.ctime().split(' ')[-2]

hms2 = hms.split(':')

h = hms2[0]

m = hms2[1]

s = hms2[2]

cv2.imwrite("camera_images/{}-{}-{}.jpg".format(h,m,s), frame)

-------------------------------

if ret == True: height, width = frame.shape[:2] data_gp = tobiiglasses.get_data()['gp'] if data_gp['ts'] > 0: cv2.circle(frame,(int(data_gp['gp'][0]width),int(data_gp['gp'][1]height)), 60, (0,0,255), 5)

# Display the resulting frame
cv2.imshow('Tobii Pro Glasses 2 - Live Scene',frame)
tobii_talker()
# Press Q on keyboard to  exit
if cv2.waitKey(1) & 0xFF == ord('q'):
  break

Break the loop

else: break

When everything done, release the video capture object

cap.release()

Closes all the frames

cv2.destroyAllWindows()

tobiiglasses.stop_streaming() tobiiglasses.close()

!/usr/bin/env python

import rospy from std_msgs.msg import String

def callback(data): rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)

def listener():

# In ROS, nodes are uniquely named. If two nodes with the same
# name are launched, the previous one is kicked off. The
# anonymous=True flag means that rospy will choose a unique
# name for our 'listener' node so that multiple listeners can
# run simultaneously.
rospy.init_node('listener', anonymous=True)

rospy.Subscriber("tobiichatter", String, callback)

# spin() simply keeps python from exiting until this node is stopped
rospy.spin()

if name == 'main': listener()

[ddetommaso]

Hi @SeungYoonLee1

thank you for your contribution! I don't have time right now to work on the ROS integration but I think it's going to be a very useful one. I think you are in the right direction to solve the issue but I was not able to see the error.

Let's keep in touch

[AmyPhung]

In case others are still interested in a ROS package, I put one together on a fork in case it's helpful: https://github.com/AmyPhung/tobii_glasses_ros. I also included a sample launch file that provides easy access to changing the ipv4 address and calibration there too

[seungyun1]

Thanks, I'll check it out!

2021년 7월 24일 (토) 오전 6:16, Amy Phung @.***>님이 작성:

In case others are still interested in a ROS package, I put one together on a fork in case it's helpful: https://github.com/AmyPhung/tobii_glasses_ros. I also included a sample launch file that provides easy access to changing the ipv4 address and calibration there too

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ddetommaso/TobiiGlassesPyController/issues/22#issuecomment-885918087, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQU32RNOSJAJ6JMJGOGIFPTTZHL2BANCNFSM4Z6GLOVA .

[seungyun1]

Hello, I don't know if you remember, but I'm a Korean university student who used Tobii on ROS last time.

Thank you so much for your help last time.

I have a question, so I'm inquiring like this.

May I know if this source code works for 'Tobii pro glasses3' as well? Thank you!

2021년 3월 31일 (수) 오전 12:07, Davide De Tommaso @.***>님이 작성:

Hi @SeungYoonLee1 https://github.com/SeungYoonLee1

good to hear new developers interested in integrating Tobii Glasses in ROS. I think it's totally doable to stream data using publishers in ROS.

For example, if you want to stream gaze position in a ROS publisher that sends string you'll simply need to encode this information as string and then pass it to the publisher.

Below you can find an example.

` import time from tobiiglassesctrl import TobiiGlassesController

import rospy from std_msgs.msg import String

def tobii_talker(): pub = rospy.Publisher('tobiichatter', String, queue_size=10) rospy.init_node('talker', anonymous=True) rate = rospy.Rate(100) tobiiglasses = TobiiGlassesController("192.168.71.50") tobiiglasses.start_streaming() time.sleep(3.0) while not rospy.is_shutdown(): tobii_gp = "Gaze Position: %s " % tobiiglasses.get_data()['gp'] rospy.loginfo(tobii_gp) pub.publish(tobii_gp) rate.sleep() tobiiglasses.stop_streaming() tobiiglasses.close() `

encode this information

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ddetommaso/TobiiGlassesPyController/issues/22#issuecomment-810339336, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQU32RNCHLCMMA5HUFLY6XDTGHSLHANCNFSM4Z6GLOVA .

[ddetommaso]

Dear @SeungYoonLee1

happy to hear from you! I have never tried the Tobii Pro Glasses 3 device, so I have really no precise idea about it. Anyway from what I get from their web page they are still using an API based on HTTP but it seems quite different from their previous API. I can bet my controller won't work with TG3!

[AkashsinhThorat]

Hello, I am a university student and a beginner in coding. I want to get gaze data from Tobii Pro Glasses 3 and publish it on rostopic in ROS. I need to get this done as early as possible to move forward with my project. It's a bit urgent!

[ddetommaso]

Hi @AkashsinhThorat

unfortunately the controller is not suitable for TG3, but only for TG2.

[AkashsinhThorat]

Thank you! Are you planning to upload any video of how you designed the controller for TG2. The thought process and explanation?

[ddetommaso]

Hi @AkashsinhThorat

I think it would be a nice idea but it's really difficult for me to take care of this task.