silexlabs / Silex

Silex is an online tool for visually creating static sites with dynamic data. With the free/libre spirit of internet, together.
https://www.silex.me
GNU Affero General Public License v3.0
2.26k stars 581 forks source link

FTP publishing method #255

Closed barkingPete closed 9 years ago

barkingPete commented 9 years ago

FTP is my preferred alternative to DropBox. SO I have set up a vsftp daemon on a Centos server running Apache. The idea is (obviously) to be able to publish (and save) to the html source directory. Which is a nice idea. There are some problems in getting Apache and vsftpd to play nicely, directory permissions, selinux policy additions and so on. However I have now got a system that works in some but not all respects. Saving and opening source files seems to work OK. But publishing seems to have some problems putting nothing (or nothing useful) in .css and .js folders. Sometimes assets are there but more recently not there.

I suppose that's a bug.

(I am willing to help debug using my setup to test results).

lexoyo commented 9 years ago

Great! You'll be my Guinea pig on this one :D I come back to you soon with a patch

lexoyo commented 9 years ago

@barkingPete I think that I have found the bug! :key:

Can you confirm that it works here please? http://preprod.silex.me/

But please note that if you edit a website with preprod, and then save it, you should not edit it with the normal version of Silex anymore...

lexoyo commented 9 years ago

For the record, here is the fix https://github.com/lexoyo/Silex/commit/b904f28de2ebf5faed7c81cf24349f87cb79bc10

barkingPete commented 9 years ago

Initial tests suggests this fix is definitely an improvement. I would say "it works!" but I must do some more thorough testing before I can say all is functioning correctly. And its Friday night..... :)

barkingPete commented 9 years ago

My first publication has produced a site with a picture and text but the text is displayed separate from the picture. It was supposed to be a picture with text superimposed over the top half. But the publication process produced a web page with "sequentially" displayed elements in the browser (Chrome). Not sure why that has happened.

I will investigate some more. I am still trying to optimise the way VSFTP server and Apache interact.

barkingPete commented 9 years ago

The published output (with layout "feature") can be viewed at www.parkadia.eu. (site is "under construction")

lexoyo commented 9 years ago

@barkingPete why does http://www.parkadia.eu/js/jquery.js says "forbidden"?

barkingPete commented 9 years ago

Yes. Good question. The problem seems to be to get a sensible setup for ftp users to be able to write directly to the /var/www/html folder while selinux is enabled - and still let apache serve the pages/scripts/images. As a "quick fix" I have altered to the folder/file permissions and this particular problem should be corrected now. I can also open, save and publish a web site without error message. So I suppose the publication process is possible - although I don't know it works correctly yet.

I have made a 3 page test site. Each page just links to the next page in a "circular" route. Each page only contains an image. When I publish and look at the page in Chrome the linking does not seem to work. I see the first page/image and that is all. Clicking on the first page does not cause it to move to the second page.

When I load and edit the page in "preprod.silex" the page navigation does not seem normal. The page navigation side bar does not work but the page navigation in the centre panel does work!?!?.

(update) preprod seems to have some issues after opening/loading a file previously saved from "preprod" - the page menu (left) "functions" strangely.

I then tried to make the teast site from new using editor.silex. When I tried to insert an image I got the file window - but it did not show my FTP folders on my test server - it showed folders on another server (or so I suppose) . The site seems to be in Polish about aerial drone photography. (drony...eu)..

That a user can possibly see someone else's data would seem to be a bit of a security issue in a multiuser system. This feature may be worth investigating by itself.

barkingPete commented 9 years ago

EDIT - Ah! I have discovered teh "log out all" button. That can be used to close the open file systems and then I can log into another FTP server. Please ignore my earlier comments on the lack of this function! :)

barkingPete commented 9 years ago

An observation about preprod.silex (30 March 2015) when I click on view or preview - I only get a message saying "saved" - similar to a file save response message. Is that a feature or am I the only one to see it?

I have made fresh test sites and published them. I have then copied the html folder from the web server to my local machine. Opening the index.html file in a browser shows a page but there is no page linking active . So publishing via FTP in preprod.silex has problems.

lexoyo commented 9 years ago

An observation about preprod.silex (30 March 2015) when I click on view or preview - I only get a message saying "saved" - similar to a file save response message. Is that a feature or am I the only one to see it?

Ouch! I see.. Thank you for the bug report Pete! I have opened a new issue with it https://github.com/silexlabs/Silex/issues/261

I have made fresh test sites and published them. I have then copied the html folder from the web server to my local machine. Opening the index.html file in a browser shows a page but there is no page linking active . So publishing via FTP in preprod.silex has problems.

Could you share the whole website with me? On dropbox maybe?

lexoyo commented 9 years ago

Big thank you @barkingPete !! Your test case was very clear, the folders and files easy to understand and complete

So I found the bug and fixed it, can you confirm (test on preprod.silex.me)? https://github.com/lexoyo/Silex/commit/630aab6899ee54ad3df25972d6bfecfe3c1404aa

barkingPete commented 9 years ago

NO! Thank you lexoyo. Breaking things is simple - I do it all the time! But fixing them is a rare skill. :0)

Anyway - Initial results were good. I made a 4 page web site and published it. I still need to sort out file masks and associated permissions for the FTP system but my first attempts were successful and can be seen on www.parkadia.org. So that is good.

However.... there seems to be a few problems still.

1- if I make and save a file in preprod.silex then try to open it - the system can "hang". It says "loading..." but nothing happens. Even if I close the browser and then reopen it again and return to preprod.silex - any attempt to open a or any file just gives me the "loading..." message on top of the file dialogue window. I cannot seem to stop this state once it has happened. I will try again in a while to see if the connection has "dropped"

2- after some work making the 4 page site with preprod.silex and saving and publishing several times to test progress - I tried to add a 5th page. I was able to do that in the editor and it seemed to publish well. But when I tried looking at the published 5 page site, I only could access/see the older 4 page site. While the new page image had been copied - the site did not show 5 linked pages. It only showed 4 pages linked as before. So the new page seems to have (somehow) been lost or not connected or something like that. I am not sure if this is related to the above problem as it happened in the same test session about the same time.

I will test more once the FTP file system has sorted itself out.

late news: I have waited over 2 hours now and the FTP file system is still "locked out". Even if I start from new and try to insert an image on page1 - clicking on "Insert" then "image" - the file dialogue still comes up with the message "loading..." and just stays in that state. There seems to be no easy way to reset this condition. So the fact that this state can be reached must be a bug some where.

barkingPete commented 9 years ago

I have discovered that deleting the associated cookies allows me to reset and use the ftp file system again. So that is useful to know. It may be that the server connection to the FTP server times out - or the ftp server disconnects after a time. That would cause a hanging connection and jam the system.

More testing of the Silex FTP system shows that in general it seems to be working. That's progress- I will experiment more with the server set up and then in due course publish the things I did to install a minimal Centos server capable of being used by Silex in "FTP mode"

hilteci commented 8 years ago

I'm seeing what may be a similar issue when publishing locally in debug mode.

I have a simple two page site with text boxes linking back to each other (preview works great). I have my publish settings configured to publish to a client/output folder. I then have nginx configured with the client/output folder as my web root. When I publish, the index.html file is written successfully, and the css js and assets directories are created successfully. All the css files are created, but they are all empty except for styles.css. The scripts.js file is not created, and all the other js files are created empty. When I point my web browser to my domain, index.html loads (so I can kind of see what the first page is sort of supposed to look like), but the site does not work or display properly without any of the css or javascript.

I've looked at the node console output when I publish, and I think the issue may be related to the data object (or maybe to path, but this only shows up in one line that I'm not sure is part of the problem). Here is my node console output:

Error in service www [TypeError: Invalid non-string/buffer chunk] TypeError: Invalid non-string/buffer chunk at validChunk (_stream_writable.js:178:14) at WriteStream.Writable.write (_stream_writable.js:205:12) at Object.exports.put (/silex/Silex/node_modules/unifile/lib/services/www.js:393:8) at doPut (/silex/Silex/node_modules/unifile/lib/core/router.js:154:21) at exec (/silex/Silex/node_modules/unifile/lib/core/router.js:165:11) at Object.exports.route (/silex/Silex/node_modules/unifile/lib/core/router.js:52:15) at Object.exports.unifileRoute (/silex/Silex/dist/server/silex-tasks.js:461:27) at Object.exports.writeFileToService (/silex/Silex/dist/server/silex-tasks.js:431:13) at IncomingMessage. (/silex/Silex/dist/server/silex-tasks.js:374:15) at emitNone (events.js:72:20) publishFiles - Error in getFile { success: false, message: 'Error in service www: TypeError: Invalid non-string/buffer chunk' } http://domain:6805/static/2.5/normalize.css /api/1.0/www/exec/put/output css/normalize.css Error in service www [TypeError: Invalid non-string/buffer chunk] TypeError: Invalid non-string/buffer chunk at validChunk (_stream_writable.js:178:14) at WriteStream.Writable.write (_stream_writable.js:205:12) at Object.exports.put (/silex/Silex/node_modules/unifile/lib/services/www.js:393:8) at doPut (/silex/Silex/node_modules/unifile/lib/core/router.js:154:21) at exec (/silex/Silex/node_modules/unifile/lib/core/router.js:165:11) at Object.exports.route (/silex/Silex/node_modules/unifile/lib/core/router.js:52:15) at Object.exports.unifileRoute (/silex/Silex/dist/server/silex-tasks.js:461:27) at Object.exports.writeFileToService (/silex/Silex/dist/server/silex-tasks.js:431:13) at IncomingMessage. (/silex/Silex/dist/server/silex-tasks.js:374:15) at emitNone (events.js:72:20) publishFiles - Error in getFile { success: false, message: 'Error in service www: TypeError: Invalid non-string/buffer chunk' } http://domain:6805/static/2.5/front-end.css /api/1.0/www/exec/put/output css/front-end.css Error in service www [TypeError: Invalid non-string/buffer chunk] TypeError: Invalid non-string/buffer chunk at validChunk (_stream_writable.js:178:14) at WriteStream.Writable.write (_stream_writable.js:205:12) at Object.exports.put (/silex/Silex/node_modules/unifile/lib/services/www.js:393:8) at doPut (/silex/Silex/node_modules/unifile/lib/core/router.js:154:21) at exec (/silex/Silex/node_modules/unifile/lib/core/router.js:165:11) at Object.exports.route (/silex/Silex/node_modules/unifile/lib/core/router.js:52:15) at Object.exports.unifileRoute (/silex/Silex/dist/server/silex-tasks.js:461:27) at Object.exports.writeFileToService (/silex/Silex/dist/server/silex-tasks.js:431:13) at IncomingMessage. (/silex/Silex/dist/server/silex-tasks.js:374:15) at emitNone (events.js:72:20) publishFiles - Error in getFile { success: false, message: 'Error in service www: TypeError: Invalid non-string/buffer chunk' } http://domain:6805/static/2.5/jquery.js /api/1.0/www/exec/put/output js/jquery.js Error in service www [TypeError: Invalid non-string/buffer chunk] TypeError: Invalid non-string/buffer chunk at validChunk (_stream_writable.js:178:14) at WriteStream.Writable.write (_stream_writable.js:205:12) at Object.exports.put (/silex/Silex/node_modules/unifile/lib/services/www.js:393:8) at doPut (/silex/Silex/node_modules/unifile/lib/core/router.js:154:21) at exec (/silex/Silex/node_modules/unifile/lib/core/router.js:165:11) at Object.exports.route (/silex/Silex/node_modules/unifile/lib/core/router.js:52:15) at Object.exports.unifileRoute (/silex/Silex/dist/server/silex-tasks.js:461:27) at Object.exports.writeFileToService (/silex/Silex/dist/server/silex-tasks.js:431:13) at IncomingMessage. (/silex/Silex/dist/server/silex-tasks.js:374:15) at emitNone (events.js:72:20) publishFiles - Error in getFile { success: false, message: 'Error in service www: TypeError: Invalid non-string/buffer chunk' } http://domain:6805/static/2.5/jquery-ui.js /api/1.0/www/exec/put/output js/jquery-ui.js Error in service www [TypeError: Invalid non-string/buffer chunk] TypeError: Invalid non-string/buffer chunk at validChunk (_stream_writable.js:178:14) at WriteStream.Writable.write (_stream_writable.js:205:12) at Object.exports.put (/silex/Silex/node_modules/unifile/lib/services/www.js:393:8) at doPut (/silex/Silex/node_modules/unifile/lib/core/router.js:154:21) at exec (/silex/Silex/node_modules/unifile/lib/core/router.js:165:11) at Object.exports.route (/silex/Silex/node_modules/unifile/lib/core/router.js:52:15) at Object.exports.unifileRoute (/silex/Silex/dist/server/silex-tasks.js:461:27) at Object.exports.writeFileToService (/silex/Silex/dist/server/silex-tasks.js:431:13) at IncomingMessage. (/silex/Silex/dist/server/silex-tasks.js:374:15) at emitNone (events.js:72:20) publishFiles - Error in getFile { success: false, message: 'Error in service www: TypeError: Invalid non-string/buffer chunk' } http://domain:6805/static/2.5/pageable.js /api/1.0/www/exec/put/output js/pageable.js Error in service www [TypeError: Invalid non-string/buffer chunk] TypeError: Invalid non-string/buffer chunk at validChunk (_stream_writable.js:178:14) at WriteStream.Writable.write (_stream_writable.js:205:12) at Object.exports.put (/silex/Silex/node_modules/unifile/lib/services/www.js:393:8) at doPut (/silex/Silex/node_modules/unifile/lib/core/router.js:154:21) at exec (/silex/Silex/node_modules/unifile/lib/core/router.js:165:11) at Object.exports.route (/silex/Silex/node_modules/unifile/lib/core/router.js:52:15) at Object.exports.unifileRoute (/silex/Silex/dist/server/silex-tasks.js:461:27) at Object.exports.writeFileToService (/silex/Silex/dist/server/silex-tasks.js:431:13) at IncomingMessage. (/silex/Silex/dist/server/silex-tasks.js:374:15) at emitNone (events.js:72:20) publishFiles - Error in getFile { success: false, message: 'Error in service www: TypeError: Invalid non-string/buffer chunk' } http://domain:6805/static/2.5/front-end.js /api/1.0/www/exec/put/output js/front-end.js

I have looked at the fixes and other references on this thread, but I can't find a directly correlating section for data.toString() for publishing locally. Is this a similar issue, or am I looking in the wrong place?

lexoyo commented 8 years ago

Hi Thank you for the bug report I have been able to reproduce and fix it

It was in teh unifile project, which is the "cloud" backend of Silex, here is the fix https://github.com/silexlabs/unifile/pull/37/

An I have updated unifile version in silex (develop branch only for now) https://github.com/silexlabs/Silex/commit/e785005ad0764b10af05e5a47a0cada3c46f59e1

To have it on your local clone, you can switch to develop branch with git checkout develop and then git pull --rebase origin develop

Tell me if I can help some more

hilteci commented 8 years ago

Thanks for the quick response, and thanks for being so willing to help. I have switched to the develop branch and updated my local clone (and manually verified the change in vim), however, I'm still seeing the same issue.

I'm still publishing to my local installation (not FTP or Dropbox), which I believe uses www.js if I'm learning the code correctly. Everything appears to publish successfully, and I can see the newly created html in my browser, but no javascript, css, or assets content is saved in my local directory, which makes the html in the browser render incorrectly.

Are there any other .toString() methods in another location which I may not have? Or is there any other information that I can provide you to better illustrate the issue I'm having?

My node console error is the same as before:

Error in service www [TypeError: Invalid non-string/buffer chunk] TypeError: Invalid non-string/buffer chunk at validChunk (_stream_writable.js:178:14) at WriteStream.Writable.write (_stream_writable.js:205:12) at Object.exports.put (/silex/Silex/node_modules/unifile/lib/services/www.js:393:8) at doPut (/silex/Silex/node_modules/unifile/lib/core/router.js:154:21) at exec (/silex/Silex/node_modules/unifile/lib/core/router.js:165:11) at Object.exports.route (/silex/Silex/node_modules/unifile/lib/core/router.js:52:15) at Object.exports.unifileRoute (/silex/Silex/dist/server/silex-tasks.js:455:27) at Object.exports.writeFileToService (/silex/Silex/dist/server/silex-tasks.js:425:13) at IncomingMessage. (/silex/Silex/dist/server/silex-tasks.js:368:15) at emitNone (events.js:72:20) publishFiles - Error in getFile { success: false, message: 'Error in service www: TypeError: Invalid non-string/buffer chunk' } http://domain:6805/static/2.5/normalize.css /api/1.0/www/exec/put/output css/normali ze.css

...with the above block repeated for each attempted file write (css/normalize.css, css/front-end.css, js/jquery.js, js/jquery-ui.js, js/pageable.js, and js/front-end.js). These files are still created as they were before, but there is still no content in them.

Thanks again - I'm still getting to know the code and learning how Silex is setup.

singchan commented 8 years ago

Please do "npm install" again