search

rickmcgeer / TransCloud

Software for the TransCloud project
5 stars 1 forks source link

Fix ultra-green cities issue #22

Open cmatthew opened 11 years ago

cmatthew commented 11 years ago

LA and some other ciites turn out to look very green. No one knows why.

cmatthew commented 11 years ago

@rickmcgeer there is a test case on greenspace.py.

rickmcgeer commented 11 years ago

@cmatthew @stredger @yyzhuang It's hanging in an ogrinfo command in trim.crop. I can run the command from the command line and it works fine, but it produces 100KB in output. So I am wondering if we are running into a situation where the process can't continue until python reads something from the pipe

rickmcgeer commented 11 years ago

The command is ogrinfo /tmp/tmpXaXw2p/tmp/19804.shp 19804, and $ ogrinfo /tmp/tmpXaXw2p/tmp/19804.shp 19804 | wc 26 2246 103554

cmatthew commented 11 years ago

Does it produce a lot of output?

there is p.communicate().

On 2013-03-09, at 5:30 PM, rickmcgeer notifications@github.com wrote:

The command is ogrinfo /tmp/tmpXaXw2p/tmp/19804.shp 19804, and $ ogrinfo /tmp/tmpXaXw2p/tmp/19804.shp 19804 | wc 26 2246 103554

— Reply to this email directly or view it on GitHub.

rickmcgeer commented 11 years ago

stackoverflow claims when you blow the buffers stuff hangs. I'm trying subprocess.check_output

yyzhuang commented 11 years ago

Which py file? Can I take a look?

rickmcgeer commented 11 years ago

trim.py. I'm playing with the copy on grack06:

/usr/local/src/greencities

yyzhuang commented 11 years ago

Just curious.. reproject_shapefile(full_shapefile, shapefile, new_projcode, shapefile_tmpDir) what's the use of full_shapefile, if you already have a parameter called shapefile?

in here: cmd = 'ogr2ogr -s_srs "EPSG:32624" -t_srs "EPSG:' + new_projcode + '" ' + new_fil + " " + full_shapefile Is EPSG:32624 the right code?

cmatthew commented 11 years ago

I dunno. I think one is used as the location of the file, the other is used to pick out the names of the rows to reproject inside the shapefile. Or something like that?

On 2013-03-10, at 3:51 PM, Yanyan Zhuang notifications@github.com wrote:

Just curious.. reproject_shapefile(full_shapefile, shapefile, new_projcode, shapefile_tmpDir) what's the use of full_shapefile, if you already have a parameter called shapefile?

in here: cmd = 'ogr2ogr -s_srs "EPSG:32624" -t_srs "EPSG:' + new_projcode + '" ' + new_fil + " " + full_shapefile Is EPSG:32624 the right code?

— Reply to this email directly or view it on GitHub.

rickmcgeer commented 11 years ago

Full shapefile is the fully qualified name of the shapefile

Sent via the Samsung Galaxy S™III, an AT&T 4G LTE smartphone

-------- Original message -------- From: Chris Matthews notifications@github.com Date:
To: rickmcgeer/TransCloud TransCloud@noreply.github.com Cc: rickmcgeer rick@mcgeer.com Subject: Re: [TransCloud] Fix ultra-green cities issue (#22)

I dunno. I think one is used as the location of the file, the other is used to pick out the names of the rows to reproject inside the shapefile. Or something like that?

On 2013-03-10, at 3:51 PM, Yanyan Zhuang notifications@github.com wrote:

Just curious.. reproject_shapefile(full_shapefile, shapefile, new_projcode, shapefile_tmpDir) what's the use of full_shapefile, if you already have a parameter called shapefile?

in here: cmd = 'ogr2ogr -s_srs "EPSG:32624" -t_srs "EPSG:' + new_projcode + '" ' + new_fil + " " + full_shapefile Is EPSG:32624 the right code?

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub.

rickmcgeer commented 11 years ago

I should really expand on this. Our previous version of the code was filling the /tmp disk with temp files generated during the computation, which was never cleaned up. To fix this bug, I built a file manager which assigned a temporary directory for ALL of the temp files generated by a worker, which could be deleted at the end of the run. To make this work, I changed the file names used through the processing pipeline to the absolute path names. However, in some cases the local name was also required (typically because it was used as a base name for various other files). In those cases, I left the base name in bnut also passed in the absolute name; this is shapefile_full.

On Sun, Mar 10, 2013 at 8:23 PM, Rick rick@mcgeer.com wrote:

Full shapefile is the fully qualified name of the shapefile

Sent via the Samsung Galaxy S™III, an AT&T 4G LTE smartphone

-------- Original message -------- From: Chris Matthews notifications@github.com Date: To: rickmcgeer/TransCloud TransCloud@noreply.github.com Cc: rickmcgeer rick@mcgeer.com Subject: Re: [TransCloud] Fix ultra-green cities issue (#22)

I dunno. I think one is used as the location of the file, the other is used to pick out the names of the rows to reproject inside the shapefile. Or something like that?

On 2013-03-10, at 3:51 PM, Yanyan Zhuang notifications@github.com wrote:

Just curious.. reproject_shapefile(full_shapefile, shapefile, new_projcode, shapefile_tmpDir) what's the use of full_shapefile, if you already have a parameter called shapefile?

in here: cmd = 'ogr2ogr -s_srs "EPSG:32624" -t_srs "EPSG:' + new_projcode + '" ' + new_fil + " " + full_shapefile Is EPSG:32624 the right code?

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/rickmcgeer/TransCloud/issues/22#issuecomment-14691370 .