Closed ashleylid closed 8 years ago
Sorry, I'm not familiar with the way matlab treats this ;) I'll see if I get the depthmap today!
using GLVisualize, FileIO, Colors, GeometryTypes, GLAbstraction
window, renderloop = glscreen()
obj = load("cat.obj")
obj_vizz = visualize(obj, color=RGBA{Float32}(0,0,0,0.1))
point3d_vizz = visualize(vertices(obj), scale=Vec3f0(0.01))
axis = visualize(boundingbox(point3d_vizz).value, :grid)
view(obj_vizz)
view(point3d_vizz)
view(axis)
@async renderloop()
yield() #needed to render first image
screenshot(window, path="color.png",channel=:color) #these are keyword arguments now :D
screenshot(window, path="depth.png",channel=:depth)
This results in these images being written out to the disk:
If you need a version which returns an Matrix{Float32}
with the depthvalue, that would be possible as well!
From what I see on Julia users, you also need to move the camera. I'll see if I can add a long needed example which shows how to do that.
You will need to do:
Pkg.checkout("GLVisualize")
to get this new change!
Okay, added the examples, rotate_robj.jl and camera.jl The first example has this output: Which I suspect is what you're after!
Thank you so much! for your speed, consistency and support.
On Tue, 24 Nov 2015 at 18:47 Simon notifications@github.com wrote:
Okay, added the examples, rotate_robj.jl https://github.com/JuliaGL/GLVisualize.jl/blob/master/examples/rotate_robj.jl and camera.jl https://github.com/JuliaGL/GLVisualize.jl/blob/master/examples/camera.jl The first example has this output: [image: catrotate] https://cloud.githubusercontent.com/assets/1010467/11373619/5858a778-92d3-11e5-8712-33b293dba45e.gif Which I suspect is what you're after!
— Reply to this email directly or view it on GitHub https://github.com/JuliaGL/GLVisualize.jl/issues/37#issuecomment-159333601 .
rotate_robj and camera are awesome! runs fine (my path messed me around). Julia and GLVisualize is the bomb!
Great to hear :) So everything is working now?
Yes!! :) And in half the time that it would have if I had been using pcl or opengl.
Sorry to be necromancing this. I am taking this depth information into a CNN, using Mocha, so I am looking to use HDF5. So instead of taking a screen shot I am looking to save the data to an hdf5 file. Is there a way to, rather than save to disk save to a high res (16bit) float array?
Sure! I could just give the option to return a Float32 array instead of saving. Would that work for you?
Yes! Thank you..
can you try if #45 works for you?
you now have screenbuffer(window, channel=:depth)
Great - I see what you did. I have updated and checked out etc but getting: ERROR: LoadError: UndefVarError: screenbuffer not defined is it the function definition?
Are you sure you checked out the branch sd/screenbuffer
?
I just used the Pkg.checkout("GLVisualize") and Pkg.update(). But I just added the code to my renderloop.jl
Ah okay, so you didn't ;). It'd have been Pkg.checkout("GLVisualize", "sd/screenbuffer")
Why do you use the rotl90(A) function?
It seems like OpenGL uses a different layout, so without rotation it comes out wrong. At least when I save it as a png.
I see. Looks like I have to force the update.
I didn't push directly to master, since I haven't tested it myself yet :D I'll merge it when I've confirmed that it works.
At least git works as expected. Stashed my changes and updating.
Perfekt, danke. Now its HDF5 territory.
If you need linear depth, here is an article describing it:
http://web.archive.org/web/20130416194336/http://olivers.posterous.com/linear-depth-in-glsl-for-real
If I read it right, this should be the formular:
z_e = 2*zFar*zNear / (zFar + zNear - (zFar - zNear)*(2*z_b -1));
You can get the znear and far from the camera
cam = window.cameras[:perspective]
cam.farclip.value
cam.nearclip.value
Me again. I would like to set my camera position and viewing angle. Looking at the camera.jl script.
My (desired) camera is sitting at this position: position = Float32[0.13053733 0.052399736 0.26690677] and its orientation is defined relatively in Euler values: (the rotation mx from this is just rotx(x), roty(y), rotz(z)) orient = Float32[1.439783 -1.0919927 -0.97914696] being [x,y,z]
I think I can load my mesh, rotate and translate it so that it corresponds to my world frame, and then set lookatvec and eyeposition to be = gripper_orient and gripper_position respectively.
My question is regarding the type. lookatvec = FixedSizeArrays.Vec{3,Float32}((-0.03494f0,-0.0349f0,-0.05762f0)) eyeposition = FixedSizeArrays.Vec{3,Float32}((0.13966f0,0.139675f0,0.211755f0))
How do I parse between Float32 array and the FixedSizeArray.Vec{3,Float} ?
If this is a basic or unrelated question please let me know and I will hop on over to the users group.
Thank you.
You mean how you convert a Vector{Float32}
to Vec{3, Float32}
?
nothing easier than that!
Vec(rand(Float32, 3))
I'll look into your other question later ;)
Oh wait, did I already fully answer your question? :D
Perhaps.. but something happened overnight.. I am getting this error:
ERROR: LoadError: TypeError: typeassert: expected Tuple{FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32}}, got Tuple{FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32}} in * at /home/ashley/.julia/v0.4/GeometryTypes/src/hyperrectangles.jl:75 in default_boundingbox at /home/ashley/.julia/v0.4/GLVisualize/src/utils.jl:62 in visualize at /home/ashley/.julia/v0.4/GLVisualize/src/visualize_interface.jl:19 (repeats 2 times) in include at ./boot.jl:261 in include_from_node1 at ./loading.jl:304 in process_options at ./client.jl:280 in _start at ./client.jl:378 while loading /home/ashley/GIT/grasp_convnet/src/input_data/generate_depth_maps_input.jl, in expression starting on line 21 [Finished in 24.6s with exit code 1]
when just running as before. line 21 is:
robj = visualize(mesh)
from camera.jl
Whoops. Well there have been changes in GeometryTypes
, but you should be
on the branch sd/typealias
, or did that change for you?
2015-12-17 11:32 GMT+01:00 kleinash notifications@github.com:
Perhaps.. but something happened overnight.. I am getting this error:
ERROR: LoadError: TypeError: typeassert: expected Tuple{FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32}}, got Tuple{FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32},FixedSizeArrays.Vec{4,Float32}} in * at /home/ashley/.julia/v0.4/GeometryTypes/src/hyperrectangles.jl:75 in default_boundingbox at /home/ashley/.julia/v0.4/GLVisualize/src/utils.jl:62 in visualize at /home/ashley/.julia/v0.4/GLVisualize/src/visualize_interface.jl:19 (repeats 2 times) in include at ./boot.jl:261 in include_from_node1 at ./loading.jl:304 in process_options at ./client.jl:280 in _start at ./client.jl:378 while loading /home/ashley/GIT/grasp_convnet/src/input_data/generate_depth_maps_input.jl, in expression starting on line 21 [Finished in 24.6s with exit code 1]
when just running as before. line 21 is:
robj = visualize(mesh)
from camera.jl
— Reply to this email directly or view it on GitHub https://github.com/JuliaGL/GLVisualize.jl/issues/37#issuecomment-165413103 .
just did my typical Pkg.update() EDIT: did a check and my branch is on sd/screenbuffer
Oh wait, mixed it up... I guess you need to get back to the right GeometryTypes commit. Try
cd GeometryTypes
git checkout master
git checkout 762f23a14528e5085d81cfb3659da321f9e85786
2015-12-17 11:41 GMT+01:00 kleinash notifications@github.com:
just did my typical Pkg.update()
— Reply to this email directly or view it on GitHub https://github.com/JuliaGL/GLVisualize.jl/issues/37#issuecomment-165416658 .
Ok cool - that worked.
position = Vec(pos) # where pos = Float32[0.13053733 0.052399736 0.26690677]
The output is:
FixedSizeArrays.Vec{3,Tuple{Float32}}(((0.13053733f0,),(0.052399736f0,),(0.26690677f0,)))
But the type of lookatvec is
FixedSizeArrays.Vec{3,Float32}((-0.03494f0,-0.0349f0,-0.05762f0))
which is what you get when I do your conversion
Vec(rand(Float32,3))
Oh, didn't realize that you use a row vector.
So either use Float32[0.13053733,
0.052399736,
0.26690677], or Vec3f0(pos...)
Oh wow.. that was my oversight. Fear I have not had enough coffee today - must remedy.
I have a question about my mesh. I would like to translate it, but I mess with the rest of the depth info when I just work with the vertices's. What I am looking to do is converting working with the vertices's and. Translating them, and then converting back to a mesh to use the depth mapping. What do you think?
Tell me if you want me to take the discussion to the users group.
what exactly is your goal, that you can't use model=translatiomatrix
?
Hi, things seem to have broken with the screen buffer and its dependencies. Did you commit this to the main branch?
EDIT: Looking through: https://github.com/JuliaGL/GLVisualize.jl/blob/master/examples/camera/camera.jl to start gathering depth maps with new protocols. What is the difference between lookatposition and eyeposition ?
It's in GLWindow now: https://github.com/JuliaGL/GLWindow.jl/blob/master/src/screen.jl#L290
Eyeposition is the position of the camera, lookat is where the camera looks at...
This is where I am:
inputs = Dict(
:window_size => Signal(Vec{2, Int}(height, width)),
:buttons_pressed => Signal(Int[]),
:mouse_buttons_pressed => Signal(Int[]),
:mouseposition => false,
:scroll_y => Signal(1)
)
lookatvec = FixedSizeArrays.Vec{3,Float32}(rot_mx * [0; 0; 1])
# move camera back 0.3 units in the -z direction of the hand
eyeposition = gripper_position[grasp_num] - (0.3 * lookatvec)
eyeposition = FixedSizeArrays.Vec{3,Float32}(eyeposition)
# create a camera from these
cam = PerspectiveCamera(inputs, eyeposition, lookatvec)
and I am getting the error: ERROR: LoadError: KeyError: scroll not found
The second example in http://www.glvisualize.com/examples/camera/ is probably the better example by now!
2016-03-07 10:30 GMT+01:00 kleinash notifications@github.com:
This is where I am:
inputs = Dict( :window_size => Signal(Vec{2, Int}(height, width)), :buttons_pressed => Signal(Int[]), :mouse_buttons_pressed => Signal(Int[]), :mouseposition => false :scroll_y => Signal(1) )
lookatvec = FixedSizeArrays.Vec{3,Float32}(rot_mx * [0; 0; 1])# move camera back 0.3 units in the -z direction of the hand eyeposition = gripper_position[grasp_num] - (0.3 * lookatvec) eyeposition = FixedSizeArrays.Vec{3,Float32}(eyeposition)
create a camera from these
cam = PerspectiveCamera(inputs, eyeposition, lookatvec)
and I am getting the error: ERROR: LoadError: KeyError: scroll not found
— Reply to this email directly or view it on GitHub https://github.com/JuliaGL/GLVisualize.jl/issues/37#issuecomment-193178781 .
Feedback
Yes great thanks - not to much different from before. I thought it had changed completely, looking at the (base) code was getting me lost.
I was looking at where PerspectiveCamera type is defined (https://github.com/JuliaGL/GLAbstraction.jl/blob/master/src/GLCamera.jl#L10) and saw that there was an update on what inputs were taken and how they where formatted Dict()
. Then there are a number of different ways to call this function (https://github.com/JuliaGL/GLAbstraction.jl/blob/master/src/GLCamera.jl#L285) (https://github.com/JuliaGL/GLAbstraction.jl/blob/master/src/GLCamera.jl#L303)
and a "new" value called upvector, what is this supposed to be?
So yah, I was lost. But the example got me back on track. Documentation is great, but I sometimes need to start a little behind what is presented. Most of the time (all) examples are really complicated for me to wrap my head around and I need to step through what is happening in the background (I started in embedded systems, so I tend to be more comfortable with first principles)
In the example (http://www.glvisualize.com/examples/camera/ ) at the end you call @async renderloop(w)
but if you just want to run it the @async
has to be removed, otherwise the window just shuts down and you don't see the image.
Issue/Question How do I set the resolution of my depth image? I want to record my depth image at a certain resolution. I have to save this in a loop for later usage with MXNet.
Bug 1
With regards to the screenbuffer function:
When I run:
img = GLWindow.screenbuffer(w, channel=:depth)
I get:
ERROR: LoadError: ArgumentError: function screenbuffer does not accept keyword arguments
So have to define channel outside of function and then pass it as an argument:
channel=:depth
img = screenbuffer(w, channel) # depth image
But color works differently
i.e.
screenshot(w; path="color$grasp_num.png", channel=:color)
> works
screenshot(w; path="color$grasp_num.png", channel=:depth)
> fails (blank image is saved)
channel=:depth; img = screenbuffer(w, channel)
> works
img = screenbuffer(w, channel=:depth)
> fails
Bug 2 There is another whole aspect to running this from the REPL. If I run it once I cant rerun the code, I have to shut down julia, rerun and reimport the function and rerun the function. Like so:
$ julia
julia> include("depthmaps.jl")
julia> depthmaps(4)
I have to shut down julia (cntl+d) and redo the above process, otherwise I get odd errors (different each time). I can run from terminal
$ julia depthmaps.jl 4
but then nothing plots, or saves because the window is not rendering.
Comment / Question
I did look at jupyter notebook - happy to use it, but I prefer sublime/vim and command line. What do you use to program?
I use Sublime and Atom ;)
Bug 2
You should instantiate the window only one time!
This workflow could be what you're after: in the REPL:
>using GLVisualize
>window = glscreen(resolution=(512,512)) # the resolution you wan the depthmap to have
> include("my_visualizations.jl") # the code you're working on, without glscreen and renderloop
>render_frame(w) # renders one frame to the screen
>screenshot(w)
>push!(camera.eyeposition, Vec3f0(...)) # update something
>render_frame(w) # save a new render
>screenshot(w)
Example for my_visualizations.jl
:
view(visualize(xxx), window)
I see if I can reproduce the screenshot bug.
upvector:
direction = position - lookat
Thanks. Haven't tried Atom, but will now.
Struggling with the code you outlined. This is how mine looks now:
using ConfParser, DataFrames, GLVisualize, GLAbstraction, FileIO, GeometryTypes, Reactive, GLWindow, PyPlot
# function to loop through output_data.csv and produce all depth maps of objects from perspective of gripper, camera distance can be modified
srand(1234)
conf = ConfParse("config.ini")
parse_conf!(conf)
# get config parameters
filtered_data_dir = retrieve(conf, "input", "filtered_data_dir")
width = retrieve(conf, "depthimages", "width", Int)
height = retrieve(conf, "depthimages", "height", Int)
num_grasps = 3
w = glscreen(resolution=(width, height))
include("rotEuler.jl")
include("transl.jl")
# read in data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
df = readtable(joinpath(filtered_data_dir, "output_data.csv"))
# describe(df)
obj_filenames = df[:objfilename]
gripper_orient = FixedSizeArrays.Vec{3,Float32}[df[:gripperorient_alpha] df[:gripperorient_beta] df[:gripperorient_gamma]]
gripper_position = FixedSizeArrays.Vec{3,Float32}[df[:gripperposition_x] df[:gripperposition_y] df[:gripperposition_z]]
# camera is 0.3m behind the hand, from the view of the hand.
cam_dis = 1
# loop through grasps - for whole file length(obj_filenames)
for grasp_num = 2:num_grasps
mesh_obj = joinpath(filtered_data_dir, obj_filenames[grasp_num])
# rot_mx spits out a Float64 [TODO: change that cause have to parse later annoying]
rot_mx = rotEuler(gripper_orient[grasp_num][1], gripper_orient[grasp_num][2], gripper_orient[grasp_num][3])
lookatvec = FixedSizeArrays.Vec{3,Float32}(rot_mx * [0; 0; 1])
eyeposition = gripper_position[grasp_num] - (cam_dis * lookatvec)
robj = visualize(GLNormalMesh(loadasset(mesh_obj)))
lookatvec = Signal(Vec3f0(lookatvec[1], lookatvec[2], lookatvec[3]))
eyeposition = Signal(Vec3f0(eyeposition[1], eyeposition[2], eyeposition[3]))
theta, translation = GLAbstraction.default_camera_control(
w.inputs, Signal(0.1f0), Signal(0.01f0)
)
upvector = Signal(Vec3f0(0,0,1))
cam = PerspectiveCamera(
theta,
translation,
lookatvec,
eyeposition,
upvector,
w.inputs[:window_area],
Signal(410f0), # Field of View
Signal(0.01f0), # Min distance (clip distance)
Signal(400f0) # Max distance (clip distance)
)
w.cameras[:my_cam] = cam
# Yes you have to view the image, OpenGL does not work unless this is done
GLVisualize.view(robj, camera=:my_cam)
#choice of saving images ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## if you want to save the color image
# screenshot(w; path="color$grasp_num.png", channel=:color)
# sleep(0.01) # let capture and save to drive
## if you want to view the depth image
# channel=:depth
# img = screenbuffer(w, channel)
# imshow(img)
# plt[:show]()
## if you want to save the depth images to some other format (this is for further work)
# saving depth images for CNN
# dset_data[:, :, 1, grasp_num] = img # store depth image
# dset_label[:, grasp_num] = grasp_measure # store grasp_measure
# empty!(w.renderlist)
# empty!(w.cameras)
end
Looking into how to find a way to update direction and position with push!
Looking into how to find a way to update direction and position with push!
Like that?
push!(camera.eyeposition, Vec3f0(...))
push!(camera.lookat, Vec3f0(...))
push!(camera.upvector, Vec3f0(...))
Yes - but now where do you define the camera? Do you perhaps loop over the push! command?
Let me try outline a little differently, I would like to:
-- load a large file with information of angle of view and camera position -- loop through sections of this large file and import a separate mesh file that will be viewed from a number of different angles and then save the depth image from that angle
wherever it fits you best? maybe at the place where you also setup the window?
Ok..
Does this work for you:
using ConfParser, DataFrames, GLVisualize, GLAbstraction, FileIO, GeometryTypes, Reactive, GLWindow, PyPlot
# function to loop through output_data.csv and produce all depth maps of objects from perspective of gripper, camera distance can be modified
srand(1234)
conf = ConfParse("config.ini")
parse_conf!(conf)
# get config parameters
filtered_data_dir = retrieve(conf, "input", "filtered_data_dir")
width = retrieve(conf, "depthimages", "width", Int)
height = retrieve(conf, "depthimages", "height", Int)
num_grasps = 3
w = glscreen(resolution=(width, height))
include("rotEuler.jl")
include("transl.jl")
# read in data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
df = readtable(joinpath(filtered_data_dir, "output_data.csv"))
# describe(df)
obj_filenames = df[:objfilename]
gripper_orient = FixedSizeArrays.Vec{3,Float32}[df[:gripperorient_alpha] df[:gripperorient_beta] df[:gripperorient_gamma]]
gripper_position = FixedSizeArrays.Vec{3,Float32}[df[:gripperposition_x] df[:gripperposition_y] df[:gripperposition_z]]
# camera is 0.3m behind the hand, from the view of the hand.
cam_dis = 1
mesh_obj = joinpath(filtered_data_dir, obj_filenames[grasp_num])
robj = visualize(GLNormalMesh(loadasset(mesh_obj)))
GLVisualize.view(robj)
# loop through grasps - for whole file length(obj_filenames)
for grasp_num = 2:num_grasps
# rot_mx spits out a Float64 [TODO: change that cause have to parse later annoying]
rot_mx = rotEuler(gripper_orient[grasp_num][1], gripper_orient[grasp_num][2], gripper_orient[grasp_num][3])
lookatvec = FixedSizeArrays.Vec{3,Float32}(rot_mx * [0; 0; 1])
eyeposition = gripper_position[grasp_num] - (cam_dis * lookatvec)
cam = w.cameras[:perspective]
push!(cam.lookat, lookatvec)
push!(cam.eyeposition, eyeposition)
push!(cam.up, Vec3f0(0,0,1))
GLWindow.render_frame(w)
img = screenbuffer(w, channel)
save("lol.png", img)
end
??
Is it possible to do something like this:
w = glscreen(resolution=(width, height))
w.cameras[:my_cam] = PerspectiveCamera()
data = load(df)
include("depthmaps.jl")
for 1:x
depthmaps(data, w)
return lookatvec; eyeposition; robj; lookatvec; theta, translation; upvector
GLVisualize.view(robj, camera=PerspectiveCamera)
end
Where then should I be asking for a screen shot?
Just checking the above..
img=screenbuffer(w,channel=:depth)
and get error: ERROR: LoadError: ArgumentError: function screenbuffer does not accept keyword argument. and it works!! :) :+1:
I have to change it so that its not loading the obj name each time. Soo will only deal with one object at a time - but this is great thank you!
Now trying to loop through all of one mesh and I just get blank output.. Yes I will totally explain what I am doing wrong - as soon as I figure it out.
Ah you have different meshes in there, I see... So you loop through camera angles and every time it's a new mesh as well?
Something like save("depth_map.png", depth_image) Even better anything that has been visualized in GLVisualize, with the similar interaction as .fig in MatLab